Q4, Q5, Q8, FP16: what quantization actually costs you

Updated 2026-08-16

Every GGUF filename carries a tag like Q4_K_M or Q8_0, and the whole local-AI world talks in them without ever explaining them. Here is what they mean and what choosing one costs you.

The one-line version

Quantization is storing each weight in fewer bits. FP16 uses 16 bits per weight. Q8 uses about 8. Q4 uses about 4. Fewer bits means a smaller file, less memory, and faster generation, at the cost of some accuracy.

That is the entire tradeoff. Everything below is detail.

The actual bit widths

The names round; the real numbers do not. From llama.cpp’s own quantize documentation, measured on Llama-3.1-8B:

Two things worth noticing. Q4 is not 4.0 bits, it is 4.89: the “K” quants mix precisions inside the file, keeping some tensors at higher fidelity. And the jump from Q8 to FP16 nearly doubles the file for a format most people running locally do not need.

The site’s what can it run page uses clean 0.5, 0.625, 1, and 2 bytes per weight, which is close enough for capacity planning and honest about being arithmetic rather than a measurement.

Smaller is also faster

This is the part people miss. Quantization is usually framed as a memory tradeoff, but generation is memory-bandwidth bound, so a smaller file streams faster. The same llama.cpp table, text generation at 128 tokens:

Q4 generates roughly two and a half times faster than FP16 on the same hardware. Those are llama.cpp’s published figures on their test setup, not ours, and your machine will differ. But the direction is structural, not incidental: half the bytes to read is half the time reading them.

Prompt processing runs the other way, mildly. FP16 leads at 923 t/s against Q4’s 822. Prefill is compute-bound, not bandwidth-bound, so shrinking the weights does not help it.

What you give up

llama.cpp is direct about it: reducing weight precision “shrinks the model’s size and can speed up inference,” but it “may introduce some accuracy loss which is usually measured in Perplexity and/or Kullback-Leibler Divergence.”

Notice what that documentation does not give: any measured perplexity number per quant type. We are not going to invent one. What is documented:

• The loss is real but reducible. It “can be minimized by using a suitable imatrix file”, an importance matrix that guides which weights keep precision.
• Requantizing is worse than quantizing once. The --allow-requantize flag carries an explicit warning that it “can severely reduce quality compared to quantizing from 16bit or 32bit”. Always start from the original weights.
• The output tensor matters disproportionately. --leave-output-tensor leaves it unquantized, which “increases model size but may also increase quality, especially when requantizing”.
• K-quants are mixtures by design. The --pure flag exists to disable “k-quant mixtures”, quantizing every tensor to one type. That the default is a mixture is the whole reason Q4_K_M outperforms a naive 4-bit format.

The practical shape of the loss, as widely reported by people running these models: it is subtle from FP16 down to Q5, noticeable but usually acceptable at Q4, and degrades quickly below 4 bits. We are stating that as the community consensus it is, not as a measurement we made.

Which one to pick

Q4_K_M is the default, and it is the right default. It is where most people land because it is the point where the file is small enough to fit interesting models on affordable hardware while the quality cost stays subtle.

Go up to Q5 or Q8 if the model already fits comfortably and you would rather have the accuracy than the speed. If you have 48 GB and want to run a 27B model, there is no reason to run it at Q4.

Go down below Q4 only when it is the difference between running a much better model and not running it at all. A 70B at Q3 often beats a 13B at Q8, because model size buys more than precision does. Below roughly 3 bits, that stops being true.

Stay at FP16 only for multimodal work or fine-tuning. llama.cpp specifically recommends high-quality formats like bf16 or q8 for multimodal encoders, since a smaller quant’s speed and memory benefit there is negligible while “overall quality could be impacted.”

The decision that actually matters

Bigger model at lower precision usually beats smaller model at higher precision. Work out what fits on what can it run, then pick the largest model that fits at Q4 rather than the smaller one you can afford to run at Q8.

Once you have picked, what can you actually run at home covers why capacity and bandwidth answer different questions, and the setup guide covers getting it running.

What is not in this guide

No perplexity or KL-divergence numbers per quant type. llama.cpp’s own documentation publishes none, and the figures circulating elsewhere are measured on specific models with specific imatrix files, which makes them poor guidance for a different model on your machine. See our methodology for why we leave those gaps rather than fill them.