Setting up llama.cpp, Ollama, and vLLM

Updated 2026-08-15

There are three tools most people end up choosing between, and they are not competing for the same job. Pick by what you are doing, not by what is popular.

llama.cpp runs quantized models on whatever hardware you have, including CPUs and Apple Silicon. It is the shortest path from nothing to a running model, and the only one of the three that works everywhere.

Ollama wraps that experience in something even simpler, with a model library and a one-line run command. Good if you want a local chat assistant and do not want to think about files.

vLLM is a serving engine. It is aimed at throughput across many concurrent requests, and it is the right answer when you are putting a model behind an API that more than one thing will call. It is also the pickiest about its environment.

If you are unsure, start with llama.cpp.

llama.cpp

The README’s quick start is two commands, depending on whether you want a chat session or a server:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The first downloads and runs a model straight from Hugging Face; the second starts an OpenAI-compatible server. Installation options listed are llama.app, Docker, prebuilt binaries from the releases page, or building from source.

Note the -hf flag doing the work there. You are not downloading a file and pointing at it. The runtime fetches the GGUF for you.

The flags that matter for memory

Two build options are worth knowing before you hit a wall.

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 allows “swapping to system RAM instead of crashing when the GPU VRAM is exhausted”. On Windows the equivalent surfaces as System Memory Fallback. Read what that promises: it prevents a crash. It is a safety net, not a speed feature, and if you are relying on it regularly you should quantize harder instead.

GGML_CUDA_FA_ALL_QUANTS compiles “support for all KV cache quantization type (combinations) for the FlashAttention CUDA kernels”, giving “more fine-grained control over KV cache size but compilation takes much longer.” Worth it if you run long contexts and are fighting for room.

Two more, same source, that people trip over: GGML_CUDA_FORCE_MMQ means “speed for large batch sizes will be worse but VRAM consumption will be lower”, and GGML_CUDA_FORCE_CUBLAS warns of possible numeric overflow with “memory use will be higher.”

On AMD with an integrated GPU, the HIP build can share main memory via UMA, but the docs warn this “hurts performance for non-integrated GPUs.” Do not enable it on a discrete card.

Ollama

The README’s chat example is one line:

ollama run gemma4

which it describes as “Run and chat with Gemma 4”. Running bare ollama prompts you to run a model or connect it to existing applications.

Worth knowing what the README does not cover, so you know where to look: it documents no serve or pull command, and states nothing about RAM or VRAM minimums or GPU selection. For those, its CLI reference and quickstart are the places to go. Any sizing advice you find elsewhere for Ollama is someone’s measurement, not a documented requirement.

vLLM

Check the prerequisites before you plan around it. The quickstart lists “OS: Linux” and “Python: 3.10 — 3.13”. If you are on macOS or Windows and want to talk to a model today, use llama.cpp.

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server is OpenAI-compatible and “By default, it starts the server at http://localhost:8000”. vLLM “automatically selects the most performant backend compatible with your system and model specifications”, overridable with --attention-backend. One gotcha the page calls out: “There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first.”

When it runs out of memory

This is where most vLLM time goes, so it is worth knowing the actual levers rather than guessing.

vLLM’s own memory guide names three ways to conserve memory:

Quantization. “Quantized models take less memory at the cost of lower precision.” Pre-quantized checkpoints from Hugging Face work with no extra setup.

Cap the context and the batch. Memory drops if you limit “the context length of the model (max_model_len option) and the maximum batch size (max_num_seqs option)”:

LLM(model="Qwen/Qwen2.5-VL-3B-Instruct", max_model_len=2048, max_num_seqs=2)

Turn off CUDA graph capture. “By default, we optimize model inference using CUDA graphs which take up extra memory in the GPU.” Either tune cudagraph_capture_sizes through compilation_config, or disable capture entirely with the enforce_eager flag.

If instead you are seeing requests get preempted, that is a different problem with different fixes. The optimization guide explains preemption happens when “KV cache space is insufficient to handle all batched requests”, and lists remedies: increase gpu_memory_utilization, since “vLLM pre-allocates GPU cache using this percentage of memory”; increase tensor_parallel_size, which “shards model weights across GPUs, allowing each GPU to have more memory available for KV cache”, with the caveat that it “may cause excessive synchronization overhead”; or lower max_num_seqs and max_num_batched_tokens.

One footgun from that same page: with chunked prefill off, max_num_batched_tokens must exceed max_model_len, or “vLLM may crash at server start-up.”

Which to install

Running a model on a laptop or a Mac, or you just want to see it work: llama.cpp. A local assistant with minimum fuss: Ollama. Serving a model to multiple concurrent clients on a Linux box with NVIDIA hardware: vLLM.

Multi-GPU changes the calculus. vLLM’s tensor_parallel_size shards parameters “across multiple GPUs within each model layer”, which the docs recommend when a model will not fit on one GPU or when you “need to reduce memory pressure per GPU to allow more KV cache space for higher throughput.” That is a real capability llama.cpp addresses differently, via hybrid CPU and GPU offload.

Before you install anything, work out whether the model fits at all. See what can you actually run at home for the arithmetic, and what can it run for the answer across every box we track.

What is not in this guide

No tokens-per-second comparisons between the three. Throughput depends on the model, quantization, context length, batch size, and runtime version, and a number from someone else’s machine tells you very little about yours. Every flag above is quoted from documentation fetched while writing this. When a version changes an interface, this guide gets updated. See our methodology.