What Is a Context Window in AI (and Why Bigger Isn't Always Better)?
Leading models read about one million tokens per request, roughly 750,000 words. Every published benchmark since 2023 shows accuracy falling as the window fills. The window is the ceiling, not the working surface.
A context window is the text, measured in tokens, that an AI model can read in a single request before it forgets. In 2026 leading models offer roughly one million tokens, about 750,000 words. The catch: every published benchmark since 2023 shows accuracy falls as the window fills. The window is the ceiling, not the working surface.
A context window is a token budget, not a memory
The model has no persistent memory of your last conversation. Each request starts blank, with one budget: the context window.
Into that budget you load everything the model needs to answer: the system prompt, the conversation so far, any documents, any retrieved data, plus space for the response. Tokens are sub-word units. OpenAI's rule of thumb is 1,000 tokens to roughly 750 English words, so a 200,000-token window holds about 150,000 words, or 300 single-spaced pages.
Hit the ceiling and the model truncates. Usually silently. The first thing to disappear is the middle of the conversation, which is exactly where the user's most recent intent lives.
The current state of the art is roughly one million tokens
As of May 2026, the frontier sits around 1M tokens for the production-tier models from the three main labs.
Anthropic ships Claude Opus 4.7 and Sonnet 4.6 with a 1,000,000-token window at standard pricing ($3 per million input tokens, $15 per million output for Sonnet 4.6). Sonnet 4.5, which carried a beta 1M window with a long-context surcharge, reverted to its 200K standard window when the beta retired on April 30, 2026. OpenAI's GPT-5.4 and 5.5 expose 1.05M tokens; GPT-5.2 sits at 400K. Google's Gemini 2.5 Pro ships at 1M, with a 2M expansion still pending; Gemini 1.5 Pro had the 2M window first.
The numbers are now close enough between vendors that "biggest window" is no longer a meaningful differentiator. What matters is what the model actually does with the tokens you pour in.
- the frontier window, about 750,000 words1M tokens
- percentage points lost when the answer sits in the middle20 to 30
- how much cheaper retrieval runs than long context8 to 82x
- a month for ten people sending 200K-token prompts$6,000
Lost in the middle dates from 2023 and is still unsolved
Liu and co-authors at Stanford and Samaya AI published "Lost in the Middle" in 2023. The finding is the headline.
They tested GPT-3.5, GPT-4, Claude, and several open models on multi-document question answering across 20 documents. Move the relevant document to position 1, accuracy is near-perfect. Move it to position 20, almost as high. Put it at position 10, accuracy collapses by 20 to 30 percentage points. A U-shape, the same shape psychologists find in human memory studies.
The U-shape persists in models with 4K, 16K, 32K, and 100K-plus windows. It is not a training gap. It is how causal attention reads.
Context rot makes the U-curve worse as the window fills
Chroma Research published "Context Rot" in July 2025: Kelly Hong, Anton Troynikov, and Jeff Huber tested 18 frontier models including GPT-4.1, Claude Opus 4, Gemini 2.5, and Qwen3.
Every model degraded as input grew. Not when the window filled. As input grew. Performance dropped at every length tested, even far below the advertised ceiling. Chroma identified three mechanisms stacking on each other: the lost-in-the-middle U-curve from Liu's work, attention dilution (transformer attention is quadratic, so 100K tokens implies 10 billion pairwise relationships; 1M tokens implies a trillion), and distractor interference, where semantically similar but irrelevant content actively misleads the model.
One Chroma result is worth memorising. Models performed better on randomly shuffled haystacks than on logically coherent documents. Adding structure that helps a human reader hurts a long-context model. That is not intuitive. It is also reproducible across all 18 models tested.
Long context and retrieval are not competitors
The 2024 industry debate framed RAG (retrieval-augmented generation) and long-context windows as rivals. The 2026 picture is hybrid.
Long-context models win on coherent, static material where the whole document needs to be in scope at once: a single contract, a single codebase under review, a single research paper being interrogated. Retrieval wins on dynamic corpora, on cost, and on signal-to-noise. A NeurIPS 2024 study from researchers at Google and the University of Michigan found long-context LLMs outperform RAG when "resourced sufficiently," but RAG remains 8 to 82 times cheaper for typical workloads. The cost gap is not closing.
In our implementations, the better question is not "RAG or long context" but "what does the model actually need to see for this turn?" The window is a budget. Spend it where the signal is.
What an SME implementer should actually care about
Three things shape what long context actually costs you, and none of them are the window size itself.
First, cost per token compounds. Sending a 200K-token prompt to Claude Sonnet 4.6 costs roughly $0.60 in input alone, every call. Multiply by ten employees, fifty calls a day, twenty working days: $6,000 a month, before output tokens. Anthropic's prompt caching, where cache reads are billed at 0.1x the base input rate, is the standard mitigation. Cache writes cost 1.25x for the 5-minute tier and 2x for the 1-hour tier, so the break-even is a single cache hit. If your tooling does not use prompt caching, you are leaving roughly 90% of the bill on the table.
Second, latency tracks input length almost linearly. A 1M-token request does not feel like a chat. It feels like a batch job. For interactive workflows, the budget is measured in seconds of patience, not millions of tokens.
Third, signal-to-noise. Every irrelevant document you dump into the window raises the chance of distractor interference. The Chroma finding holds: more is not safer. Curate.
The takeaway is unsentimental. Longer windows do not replace good information architecture. They raise the ceiling on what you can attempt. The work of deciding what the model actually needs to see, of structuring it so the model can find the answer, of routing the right system to the right query, that work does not get easier as windows grow. It gets more important, because the cost of being sloppy is now invisible in the output rather than visible in a truncation error.
A context window is the room. What goes in it is the more interesting question. For how AI systems get access to your actual business data in the first place, see What Is MCP and Why It Matters for Your Business?
What is a context window in AI in simple terms?
A context window is the maximum amount of text an AI model can read in a single request, measured in tokens. Tokens are sub-word units; 1,000 tokens equals roughly 750 English words. Everything the model needs to answer (instructions, conversation history, documents, room for the response) has to fit inside that budget.
How big are the context windows of GPT-5, Claude, and Gemini in 2026?
As of May 2026, Claude Sonnet 4.6 and Opus 4.7 offer 1,000,000-token windows at standard pricing. OpenAI's GPT-5.4 and GPT-5.5 expose 1,050,000 tokens; GPT-5.2 stops at 400,000. Google's Gemini 2.5 Pro ships at 1,000,000. Gemini 1.5 Pro held a 2,000,000-token window, the largest yet released to general availability.
What is 'lost in the middle' and does it still happen?
Lost in the middle is the 2023 finding by Liu et al. (Stanford and Samaya AI) that LLMs recall information at the start or end of a long context but miss it when it sits in the middle. The accuracy drop is 20 to 30 percentage points. Chroma's July 2025 study across 18 frontier models confirms the pattern persists.
Is a bigger context window always better?
No. Chroma Research's July 2025 study tested 18 frontier models and found performance degrades at every input length increment, well below the advertised ceiling. The mechanisms are architectural (quadratic attention, U-shaped position bias, distractor interference), not training gaps. A focused 10,000-token prompt routinely outperforms a kitchen-sink 500,000-token prompt on the same task.
How does long context compare to RAG for business use?
Long-context windows suit static, coherent material where the whole document matters at once. RAG (retrieval-augmented generation) suits dynamic corpora and runs 8 to 82 times cheaper per query in typical workloads. The 2026 answer is both: retrieval narrows what the model sees, the window holds what it needs. Pick by cost, latency, and source volatility.