AI Gear Watch

Multi-GPU: tensor parallel, layer split, and when NVLink matters

Updated 2026-08-15

Adding a second GPU solves one problem well and a different problem badly.

It solves capacity. Two 24 GB cards hold a model that will not fit in one. That is the reason most people do it, and it works.

It does not automatically solve speed. Whether the second card makes generation faster depends entirely on how you split the model and how the two cards talk to each other. Split it the easy way and the GPUs take turns rather than working together.

The two ways to split

By layer (pipeline). Layers 0 to 39 live on GPU 0, layers 40 to 79 on GPU 1. A token passes through GPU 0, then GPU 1. llama.cpp calls this layer and documents it as “split layers and KV across GPUs (pipelined)”. It is llama.cpp’s default.

This is the easy mode. Communication between cards is tiny, just the activations handed from one stage to the next. But for a single request the GPUs are mostly taking turns: while GPU 1 works, GPU 0 waits. You get the combined memory, not the combined throughput.

By tensor (parallel). Every layer is split across both cards, and they work on it simultaneously. llama.cpp’s tensor mode “split weights and KV across GPUs (parallelized, EXPERIMENTAL)” does this, as does vLLM’s tensor_parallel_size, which vLLM describes as sharding parameters “across multiple GPUs within each model layer” following Megatron-LM’s algorithm.

This is what actually uses both cards at once. The cost is that they must synchronize inside every layer, many times per token. That traffic is where your interconnect starts to matter.

Here is the answer, and it comes from vLLM’s own guidance rather than folklore:

“if the GPUs in your node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead” (vLLM parallelism guide)

So NVLink matters precisely when you want tensor parallelism. Without a fast link between cards, tensor parallel’s per-layer synchronization costs more than it gains, and the layer split is the better choice even though it uses the cards less fully.

That inverts the usual shopping instinct. If your two cards are connected only by PCIe, do not reach for tensor parallel and then wonder why it is slow. Use pipeline mode, accept that you bought capacity rather than speed, and spend the money you would have spent on interconnect on more memory instead.

vLLM adds a second reason to prefer pipeline: it “splits the model along layers and supports uneven splits”, which tensor parallel does not. Mismatched cards are a pipeline job.

Picking a configuration

vLLM’s guidance is refreshingly blunt: if the model fits on one GPU, “distributed inference is probably unnecessary”. Check that first on what can it run before you buy anything.

Beyond that, the documented ladder is:

After starting vLLM, read the startup log for GPU KV cache size and Maximum concurrency. Those tell you whether you actually have room to serve, and the concurrency estimate derives from max_model_len. If they are lower than you need, that is the signal to add hardware rather than tune further.

The llama.cpp flags

Four flags control this, all from the server README:

-sm, --split-mode takes none, layer, row, or tensor. none uses one GPU only. row splits “weight across GPUs by rows (parallelized)”. tensor is marked EXPERIMENTAL, so treat it accordingly.

-ts, --tensor-split sets the “fraction of the model to offload to each GPU, comma-separated list of proportions, e.g. 3,1”. This is the flag for mismatched cards: a 24 GB and an 8 GB card want roughly 3,1, not an even split.

-mg, --main-gpu picks which card holds the model under split-mode = none, or holds intermediate results and KV under split-mode = row.

-ngl, --n-gpu-layers is the “max. number of layers to store in VRAM, either an exact number, ‘auto’, or ‘all’”, defaulting to auto. There is also -fit, --fit to auto-adjust arguments “to fit in device memory”, and --list-devices to see what the runtime can actually see, which is the first thing to run when a card is missing.

Splitting across machines

llama.cpp can do this with ggml-rpc-server, which “allows exposing ggml devices on a remote host”. Build with -DGGML_RPC=ON, run the server on each remote box, and point the main host at them with --rpc 192.168.88.10:50052,192.168.88.11:50052. Weights and KV cache spread across local and remote devices “in proportion to each device’s available memory”, overridable with --tensor-split.

Read the project’s own warning before you try it. This is “a proof-of-concept development stage” where “the functionality is fragile and insecure”, and the docs say plainly: “Never run the RPC server on an open network or in a sensitive environment!” That is the maintainers talking about their own feature, not us being cautious on their behalf.

One practical tip from the same page: the local cache flag -c stores large tensors locally to avoid re-transferring them, which “can speed up model loading significantly, especially when using large models”.

vLLM’s multi-node path is Ray or its own multiprocessing launcher, and it requires an “identical execution environment, including the model path and Python packages” on every node. Containers are recommended for exactly that reason. Its networking guidance also carries a warning worth repeating: cross-node traffic is “unencrypted”, so keep it on a private network.

If you have InfiniBand, verify you are actually using it. Set NCCL_DEBUG=TRACE and look at the transport line. [send] via NET/IB/GDRDMA means RDMA is working; [send] via NET/Socket is, in vLLM’s words, “not efficient for cross-node tensor parallelism”. Silently falling back to sockets is the classic multi-node performance bug.

What to buy

Two thoughts that follow from all of the above rather than from benchmarks.

More memory on one card beats the same memory across two, when you can afford it. One card has no split to configure, no interconnect to bottleneck, and no experimental flags. Compare the single-card options before assuming you need two.

If you are buying two, buy identical cards. Tensor parallel wants them matched, and mismatched cards push you into pipeline mode with a hand-tuned --tensor-split ratio.

What is not in this guide

No speedup numbers. Scaling depends on the model, the split mode, the interconnect, the batch size, and the runtime version, and a ratio from someone else’s rig tells you very little about yours. Everything above is quoted from documentation fetched while writing this. See our methodology, and start with what can you actually run at home if you have not worked out whether you need a second card at all.

All guides