Blog · Hardware Basics · September 18, 2026 · 8 min read
How Much VRAM Do You Actually Need? Context, Quantization, and the Math Nobody Explains
Every “can my GPU run this?” thread on the internet gives you a vibe-based answer. The truth is that VRAM requirements come from exactly two places — the model weights and the context cache — and both are predictable. This post gives you the actual math, then shows where our compatibility panel automates it for you.
Part 1: The weights (the big, fixed cost)
Every parameter of the model must sit in VRAM, and the precision it is stored in decides how many bytes each parameter eats:
| Precision | Bytes / parameter | 8B model weighs |
|---|---|---|
| FP16 (full precision) | 2.0 | ~16 GB |
| Q8 | 1.0 | ~8 GB |
| Q4 | 0.5 | ~4 GB |
| Q2 | 0.25 | ~2 GB |
The shortcut: parameter count × bytes, then divide by 1024³ for GiB. That is the entire weights calculation — it does not change with your prompt, your conversation length, or the moon phase.
Part 2: The KV-cache (the growing cost)
While generating text, the model stores a running memory of every token in your context — the KV-cache. It grows linearly with context length and depends on the model’s architecture (depth and how narrowly it caches attention):
KV bytes = 2 × layers × kv_dim × 2 bytes × context_tokensTwo things surprise people here. First, it is linear — doubling your context doubles this cost. Second, it varies wildly per model: an 8B model with grouped-query attention (Llama 3.1 8B) costs about 0.12 GB per 1,000 tokens, while an older full-attention design of the same size can cost four times that. This is exactly why we precompute a kvGBPer1kTokens coefficient per model instead of guessing.
Worked example: Llama 3.1 8B
| Configuration | Weights | Context cache | Total |
|---|---|---|---|
| FP16, 4K context | 14.96 GB | 0.5 GB | 15.5 GB |
| Q4, 8K context | 3.74 GB | 1.0 GB | 4.7 GB |
| Q4, 32K context | 3.74 GB | 3.9 GB | 7.7 GB |
| Q4, 128K context | 3.74 GB | 15.6 GB | 19.3 GB |
Notice what happened in the last row: at 128K context the cache is four times the size of the weights. Context length is often the real reason a model “suddenly” stops fitting.
Part 3: The headroom rules
- Never plan to use 100% of VRAM. Your operating system and display take 0.5–2 GB, and inference frameworks need scratch space. We flag a model as an Excellent Fit only when it uses ≤85% of your VRAM.
- Multi-GPU adds capacity, not speed. Two 16 GB cards give you ~32 GB of capacity, but bandwidth — and therefore tokens per second — does not double.
- Leave room to grow the context. If you plan to feed documents, budget the cache for your realistic document size, not the slider minimum.
Skip the math entirely
Pick your card in the Hardware panel, drag the context slider, and read the fit badges — the same formula above, computed for all 90 models in the library at once. The green “Excellent Fit” badge is the 85% rule; the yellow “Tight Fit” badge means it will run but with little room to spare.