Paste a forty-page contract into a chatbot, ask a question about clause twelve, and sometimes you get an answer that quietly ignores what clause three said. The model didn’t run out of room. It had the whole document in front of it. It just didn’t use all of it evenly, and that distinction is the entire story of what a context window actually is and isn’t.
Every AI vendor now advertises context windows in the hundreds of thousands or millions of tokens, and the number keeps climbing. What rarely gets explained is why a bigger number doesn’t automatically mean the model understood your document better, or why two models with an identical advertised limit can behave completely differently on the same 50-page file.
What Is a Context Window?
A context window is the maximum amount of text, measured in tokens rather than words, that a model can process in a single request – including your prompt, any documents you attach, the conversation history, and the reply it generates. Once a conversation or document exceeds that limit, older content has to be dropped, summarized, or excluded before the model can respond at all.
Tokens are the unit that makes this countable. A token is a chunk of text, not a word: short, common words like “the” usually become one token, while rarer or longer words split into pieces. OpenAI’s own documentation puts the rough conversion at one token per four characters, or about 0.75 words of English text – so a 1,000-word memo runs to roughly 1,300 tokens once you count the system prompt and formatting around it.
Why “1 Million Tokens” Doesn’t Mean What It Sounds Like
The industry has moved fast on this number. Several frontier models now advertise context windows at or beyond one million tokens, which sounds like it should hold an entire novel, a full legal case file, or a mid-sized codebase in one request. Technically, it can.
Practically, holding that much text and using all of it correctly are different engineering problems, and the second one is far from solved.
The U-Shaped Recall Problem
In 2023, a team of researchers at Stanford and UC Berkeley published a study that has since become one of the most cited papers in applied AI: Lost in the Middle: How Language Models Use Long Contexts. The researchers fed models long documents with a specific fact placed at varying positions, then asked questions that could only be answered using that fact.
The result was a consistent U-shaped curve. Models were reliably accurate when the answer sat near the beginning or end of the input. Accuracy dropped, sometimes sharply, when the same fact sat in the middle of a long context – even in models explicitly built and marketed for long-context use. The paper’s authors found this held across the model families they tested, not as an occasional glitch but as a structural pattern in how these systems attend to their own input.
This is not a settled, permanently-fixed problem. Retrieval techniques, better training data, and architectural changes have all narrowed the gap since 2023. But the underlying tendency – treat the start and end of an input as more important than the middle – has reappeared in newer benchmarks on models with much larger windows, which suggests it’s a byproduct of how these models are trained rather than a limitation tied to any one model generation.
Advertised vs. Effective Context
This gap between the number on the spec sheet and what the model reliably uses has earned its own shorthand among researchers: advertised context versus effective context. Advertised context is the technical ceiling the vendor will accept as input. Effective context is the point past which recall accuracy measurably degrades on realistic tasks.
The two are rarely the same number, and the gap tends to widen as the advertised ceiling grows. A model that holds 200,000 tokens with strong recall throughout is often more useful for a document-heavy task than one that holds a million tokens but only reliably tracks the first and last fifth of it.
How Much Context You Actually Need
Most real work needs far less than the advertised maximum. Before choosing a tool based on its context window, it helps to know roughly where your own task sits.
| Task | Approximate tokens needed |
|---|---|
| A short email or single-page memo | 200–500 |
| A typical blog post or article draft | 1,500–3,000 |
| A 20-page PDF report | 15,000–20,000 |
| A novel-length manuscript | 100,000–150,000 |
| A mid-sized codebase (a few hundred files) | 200,000–500,000 |
| Years of accumulated meeting notes or logs | 1,000,000+ |
Anyone working mainly with single documents under about 50 pages rarely needs to think about context window size at all – nearly every current model comfortably covers that range. The number starts to matter once you’re feeding in entire codebases, multi-document research corpora, or long accumulated histories in a single request.
Getting More Out of a Limited Window
Regardless of which model you use, a few habits make better use of the space you have:
- Put the critical instruction at the start or the end of the prompt, not buried in the middle of a long pasted document, since that’s where recall is strongest.
- Trim documents to the relevant sections before pasting them in, rather than relying on the model to find the needle in a much larger haystack.
- Break large tasks into smaller passes – summarize sections first, then reason over the summaries – instead of demanding one shot over the entire input.
- Re-state key facts near the end of a long prompt if they matter for the final answer, since recency helps recall.
- Check the vendor’s own documentation for the model’s actual context limit, not a comparison chart, since these numbers change with almost every release.
Frequently Asked Questions
Does a bigger context window always mean better answers? No. A larger window increases what a model can technically accept, but recall accuracy on the middle of long inputs has been shown to lag behind recall at the edges, so size alone doesn’t guarantee the model used everything you gave it.
How many words is 100,000 tokens? Using the roughly 0.75-words-per-token conversion, 100,000 tokens works out to approximately 75,000 words, or a moderately long novel.
Does context window size affect the cost of a request? Yes. Nearly every provider bills by the token, so a longer input costs more regardless of whether the model uses all of it effectively. The mechanics of that pricing are covered in our guide to how AI token pricing works.
Why does the same model sometimes seem to “forget” something I told it earlier? If the conversation has grown long enough to approach the context limit, older messages may have been dropped or summarized to make room for new ones. This is a common cause of a model appearing to contradict something said earlier in a long session.
Is a 1-million-token window the same across every provider? No. Providers measure and price tokens differently, and advertised maximums don’t tell you how well a model performs at that length. Always check a provider’s current documentation rather than relying on a comparison chart, since these figures change often.
Can I just always use the model with the largest context window? You can, but it isn’t automatically the best choice. Larger windows often carry a price premium and don’t guarantee better recall. Matching the window to your actual task, using the table above as a starting point, is usually more useful than defaulting to the biggest number available.
The Practical Takeaway
Context window size tells you what a model will accept, not what it will reliably use. When we test a tool’s ability to handle long documents as part of our review methodology, we check recall at different positions in the input rather than trusting the advertised maximum on its own – the same U-shaped pattern researchers documented in 2023 is still worth checking for in any model you’re evaluating today. If a task genuinely depends on a fact buried in the middle of a long document, the safest habit is still to test it directly rather than assume a bigger number solves the problem. For more on the mechanics driving this and related model behavior, see our coverage under artificial intelligence, including our analysis of why AI models hallucinate in the first place.