Blog · Concepts · September 20, 2026 · 6 min read
Tokens Per Second Explained: Why Memory Bandwidth Beats Everything
.webp)
When a model writes an answer one token at a time, the bottleneck is not the GPU’s raw compute — it is how fast the whole model can be read from memory. Generating each token requires streaming the (active) weights through the chip, which gives local AI’s most useful formula:
tokens per second ≈ memory bandwidth ÷ model size in bytesExample: an RTX 4090 moves 1008 GB/s, and a Q4-quantized 8B model is about 4.5 GB — so a top speed of roughly 200+ tokens per second on paper, with real-world chat landing lower once overheads bite. The same model on a card with 256 GB/s of bandwidth lands near 50. The estimate shown for each model in our compatibility panel is exactly this calculation for your selected card.
What the numbers feel like
| Speed | Experience while chatting |
|---|---|
| ~5 t/s | Painful — reads like slow typing; usable only for short answers |
| ~15 t/s | Acceptable — noticeably slower than reading speed |
| ~30 t/s | Comfortable — roughly reading pace |
| 60+ t/s | Instant-feeling; long answers stream like a fast typist |
Reading speed in English is around 5–7 words per second, and a token is about ¾ of a word — so ~15–20 t/s matches human reading. Anything faster is nice; anything slower feels like waiting.
Why a gaming champion loses at AI
Consider two cards: one with blistering compute cores but modest memory bandwidth, and one that is older and slower at games but has wide memory (an RTX 3090 at 936 GB/s, say). For frame rates, compute wins. For token generation, the wide-memory card wins by a mile — it can simply feed the model faster. This is why VRAM bandwidth is the first spec to check after capacity, and why our budget GPU comparison ranks cards by bandwidth per dollar.
Fine print worth knowing
- Prefill vs decode: reading your prompt (prefill) is compute-heavy and nearly instantaneous; the t/s figure governs the answer streaming out (decode). Very long prompts can still take a moment to digest on slow cards.
- Quantization helps twice: a Q4 model is not just a quarter of the memory — being smaller, it is also read faster, so tokens per second roughly quadruple versus FP16 on the same card.
- MoE models are speed cheats: with only a fraction of experts active per token, less memory is read per step — that is why a “235B” can feel like a 22B (see our MoE explainer).