What can you actually run at home?

Updated 2026-08-15

Two numbers decide almost everything about running models locally, and neither of them is the one on the box.

Memory capacity decides whether a model loads at all. If the weights plus the working state do not fit, nothing else matters: you either fall back to much slower paths or you do not run the model.

Memory bandwidth decides how fast it answers. Generating a token means reading essentially the whole model out of memory once. That is why a card with enormous compute and modest bandwidth feels slower than its spec sheet suggests, and why a Mac with unremarkable compute but very wide memory does better at this than people expect.

Compute matters too, but it is the third question, not the first.

Step one: will it fit?

Take the parameter count, multiply by the bytes per weight for your quantization, then leave headroom.

llama.cpp supports “1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use”. Bits per weight is exactly what the name says: 4-bit is roughly half a byte per parameter, 8-bit roughly one byte, and unquantized FP16 two bytes.

So a 70B model is roughly 35 GB of weights at 4-bit, 70 GB at 8-bit, and 140 GB at FP16. Our what can it run table does this arithmetic across every box we track, and adds a 20 percent allowance on top for the KV cache, activations, and the runtime itself.

Be honest about that allowance. It is a planning margin we chose, not a measurement, and the real overhead grows with context length and batch size. A model that fits with 2k of context may not fit with 64k.

What happens when it does not fit

You have three options, and they are not equally good.

Quantize harder. Going from 8-bit to 4-bit roughly halves the weights. This is the first thing to try and usually the right answer. Quality degrades as you go down, and it degrades faster below 4-bit than above it.

Offload part of the model. llama.cpp supports “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity”. This works, and it is dramatically slower than staying in VRAM, because the layers living in system RAM are read across a much narrower bus. Offloading a few layers is a reasonable compromise. Offloading half the model is usually a sign you should quantize harder instead.

Let it spill automatically. On CUDA you can set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1, which llama.cpp’s build docs describe as allowing “swapping to system RAM instead of crashing when the GPU VRAM is exhausted”. Read that sentence carefully: the feature it advertises is not crashing. It is a safety net, not a performance strategy.

Why bandwidth is the number to compare

Once the model fits, generation speed is governed by how fast the hardware can stream weights out of memory. This is why the bandwidth column on our hardware list is worth more of your attention than the compute column.

It is also why the interesting comparison is not always the obvious one. An RTX 5090 has 32 GB at 1792 GB/s. A Mac Studio with M3 Ultra has 512 GB at 819 GB/s. The Mac has less than half the bandwidth and sixteen times the capacity. For a model that fits in 32 GB the 5090 wins on speed. For a model that needs 200 GB the 5090 does not enter the conversation at all.

Compare them directly if you want to see the tradeoff laid out.

Unified-memory machines sit at one end of this spectrum: large pools, moderate bandwidth. Discrete GPUs sit at the other: small pools, very high bandwidth. Neither is universally better. The question is which side of the fit-versus-speed tradeoff your workload lands on.

Where the memory actually goes

Weights are the big number, but they are not the only one.

The KV cache grows with context length and with batch size, and it lives in the same memory as the weights. It can be quantized too: llama.cpp’s build documentation describes a GGML_CUDA_FA_ALL_QUANTS option that compiles “support for all KV cache quantization type (combinations) for the FlashAttention CUDA kernels”, noting it gives “more fine-grained control over KV cache size but compilation takes much longer.” If you are running long contexts and running out of room, that is a lever worth knowing about.

We are deliberately not printing a KV-cache sizing formula here. The ones circulating are model-architecture-specific, and we have not verified one against a source we trust. Measure yours rather than trusting a number from a forum post, including this one.

Getting something running

The fastest path from nothing to a running model is llama.cpp. Its README documents the quick start as llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF, which downloads and runs a model straight from Hugging Face, or llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF for an OpenAI-compatible server.

vLLM is the other common choice, aimed at serving rather than single-user chat. Its quickstart lists prerequisites of “OS: Linux” and “Python: 3.10 — 3.13”, installs with uv pip install vllm --torch-backend=auto, and serves with vllm serve Qwen/Qwen2.5-1.5B-Instruct. If you are on macOS or Windows and just want to talk to a model, start with llama.cpp instead.

The rule we hold ourselves to

You will notice this guide contains no tokens-per-second figures. That is deliberate. Throughput depends on the model, the quantization, the context length, the batch size, the runtime version, and the specific machine, and a number copied from someone else’s setup tells you very little about yours.

What we will tell you is what fits, because that follows from capacity, which is a published spec we can trace to a vendor page. Everything on the hardware pages links to where it came from. See our methodology for how that works.

When you want a real throughput number, measure it on your own hardware with your own model. That number is worth more than any benchmark table.