Blog · Concepts · September 20, 2026 · 7 min read

Context Windows Explained: What 8K vs 128K Actually Means for You

A model’s context window is its working memory: the total amount of text — your instructions, the whole conversation so far, and any pasted documents — that the model can consider at once. When people say a model has “8K context,” they mean roughly 8,000 tokens, and tokens are not words: for English text, one token is about four characters or three-quarters of a word.

What actually fits

Everything counts against the limit: the system prompt, every message you sent, every reply the model gave, and the entire pasted document. Long conversations do not just risk hitting the ceiling — once the window is full, older turns fall out of memory entirely, which is why a chatbot sometimes “forgets” what you said an hour ago.

Two hidden costs of long context

• Memory grows linearly. The KV-cache that tracks your context lives in VRAM, and on a typical 8B model it costs roughly 0.12 GB per thousand tokens. At 8K that is ~1 GB — at 128K it is ~16 GB, possibly more than the model weights themselves. Drag the context slider in our compatibility panel and watch the numbers move.
• Attention quality fades. Even when a document fits, models are noticeably better at using information in the middle of a huge context than marketing suggests. Retrieval — handing the model only the relevant pages — often beats raw context stuffing (see our RAG explainer).

Practical guidance

• Chatting: 8K–16K is plenty for conversations that get summarized or reset.
• Working with one or two documents: 32K is a comfortable zone.
• Whole-book analysis: 128K-class contexts work, but budget the VRAM for it — and prefer retrieval if the task is search-like.

The bottom line: pick context length per task, not as a badge. The maximum a model supports and the maximum your VRAM supports are different numbers — the slider on this site shows the gap for every model in the library.