Blog · Concepts · September 19, 2026 · 8 min read
Q4 vs Q8 vs FP16: What Quantization Actually Costs You in Quality and Memory
.webp)
Quantization is the reason a 70-billion-parameter model can run on a single gaming GPU. It stores each of the model’s weights using fewer bits — and the trade it makes is a little precision in exchange for a lot of memory. Understanding the levels is the difference between choosing a model deliberately and downloading blind.
The levels, plainly
| Level | Bits / weight (approx.) | Memory vs FP16 | Quality character |
|---|---|---|---|
| FP16 | 16 | 100% | The reference — what the model was trained to be |
| Q8 | 8 | 50% | Nearly indistinguishable from FP16 for most uses |
| Q4 | 4–4.5 | ~25% | The practical sweet spot — small real-world quality loss |
| Q2 | 2–2.5 | ~12.5% | Noticeably degraded; a last resort for huge models |
You will also see suffixes like Q4_K_M or Q4_K_S on GGUF files (the format Ollama uses). The K variants quantize especially sensitive layers more carefully; _M (medium) is the usual recommendation and what Ollama downloads by default.
What quality loss actually looks like
- Q8: benchmarks move by fractions of a percent. If it fits at Q8, there is rarely a reason to keep FP16 around.
- Q4: the community’s default. Factual recall and reasoning stay strong; the most visible degradation is in long, nuanced generations and rare edge cases. For chat, coding help, and summarization it is usually the right call.
- Q2: where models start losing coherence on multi-step reasoning and niche knowledge. Useful when the alternative is “doesn’t run at all” — for example, fitting a 70B into 20 GB. Treat outputs with suspicion.
One rule deserves bold text: smaller models suffer more from aggressive quantization than larger ones. A Q4 70B often outperforms an FP16 13B despite being “more compressed,” because base capability matters more than precision. When in doubt, a bigger model at Q4 beats a smaller model at Q8.
The memory arithmetic
The bytes-per-parameter numbers above are the whole story for the weights: an 8B model at FP16 needs ~16 GB, at Q8 ~8 GB, at Q4 ~4 GB. What the weights do not include is the context cache, which is unaffected by weight quantization (it is kept at higher precision) and grows with your conversation length. That is why a Q4 8B model that fits at 4K context can still run out of memory at 64K.
A small honesty note about our estimates: our compatibility panel uses clean ratios (25% for Q4 and so on). Real GGUF Q4 files land slightly above that (~0.55 bytes/parameter once metadata is included), so treat the Q4 row as slightly optimistic and keep your headroom — which is also why the “Excellent Fit” badge requires 15% free VRAM.
A decision matrix
- Model fits at Q4 with room to spare? Use Q4 — speed and capacity win.
- Chasing maximum quality on a big-VRAM card? Q8 is a luxury that costs little to keep.
- Only Q2 fits? First check a smaller sibling model at Q4 — it will usually be the better experience.
- Everything fits easily? You are probably bandwidth-rich; try a larger model at Q4 rather than a small one at FP16.