What Is a Context Window in AI?

A context window is the maximum amount of text - measured in tokens - that a large language model can process in a single request. Everything the model uses to generate its response, including the system prompt, the conversation history, any retrieved documents, and the user's question, must fit inside this window. When the limit is reached, older content is dropped, truncated, or summarized.

The context window is one of the most consequential design parameters of any LLM-based application. It determines how much background a model can hold while it reasons, how many documents a RAG system can retrieve at once, and how many turns a conversation can run before the model loses track. Frontier model windows have grown from the 2,048 tokens of GPT-3 (2020) to 10 million tokens for Meta's Llama 4 Scout (2025) - nearly a 5,000x increase in five years.

What Is a Context Window?

Every LLM is a fixed-size machine: it accepts a defined number of tokens as input and produces tokens as output, one at a time. The context window is the cap on input + output combined (or sometimes split, depending on the provider). The window represents the model's working memory - but unlike human memory, it is reset to zero at the start of each request. Anything the model "remembers" from earlier in a session is only remembered because the application replayed it back into the prompt.

Context windows differ from the broader concept of model memory. A context window is short-term and per-request. Persistent memory, by contrast, is something the application layer adds - by storing past conversations, summarizing them, and re-injecting relevant pieces into each new prompt. The context window is the bottleneck through which every piece of that memory must pass.

How Tokens Work

LLMs do not see text - they see tokens. A token is a chunk of text produced by a tokenizer, the algorithm a model uses to break input into manageable pieces. Tokens roughly correspond to common substrings: short words ("the", "and") are usually one token; longer or rare words split into multiple tokens. Punctuation, whitespace, and Unicode characters all consume tokens too.

A useful rule of thumb for English text, with the caveat that the exact ratio varies by tokenizer and model. Google puts 100 Gemini tokens at about 60-80 English words, and Anthropic says 1M tokens is roughly 555,000 words on its current Claude tokenizer:

• 1,000 tokens ≈ 550 to 800 English words or roughly one to two pages of plain prose
• 128,000 tokens ≈ 70,000 to 100,000 words or a book of a few hundred pages
• 1,000,000 tokens ≈ 550,000 to 800,000 English words, depending on the tokenizer, or roughly the length of War and Peace at the low end
• 10,000,000 tokens - Llama 4 Scout's window - covers roughly 5.5 to 8 million words, the equivalent of a small library

Non-English languages, code, and structured data tokenize differently. Czech, German, and other morphologically rich languages often need more tokens for the same meaning. Source code is often more efficient (1 token can represent a full symbol like function), but heavy whitespace or unusual characters can blow up the count. Always check token usage with the provider's tokenizer before designing prompts that rely on tight limits.

Context Window Sizes Across Models (2026)

As of October 2026, frontier context windows fall into four tiers. Lineups change quickly, so check the vendor documentation before you design around a specific limit. The numbers below reflect the maximum advertised window - actual useful context is usually smaller (see Lost in the Middle).

Tier 1 - 10M tokens

Meta Llama 4 Scout, released in April 2025, has a 10-million-token context window, one of the largest published windows. At this size, the model can hold an entire mid-sized codebase or a year of email history in a single request. Real-world utility at the upper end is debated - see the cost and "lost in the middle" sections below.

Tier 2 - about 1M tokens

The 1-million-token tier is the frontier mainstream. Anthropic's current models, Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Haiku 5.5, all have 1M-token windows with up to 128K output tokens. Google's Gemini 3.1 Pro (preview) and Gemini 3.8 Flash accept 1,048,576 input tokens and up to 65,536 output tokens. OpenAI's GPT-6.1 Sol has a 1,050,000-token window (up to 922,000 input tokens) with 128K output, priced higher above 272K input tokens. Llama 4 Maverick supports 1M tokens.

Tier 3 - 128K tokens

The 128K tier is now mostly older or smaller models. GPT-4 Turbo (2023), which OpenAI is scheduled to shut down on October 23, 2026, GPT-4o, Llama 3.1 405B, and Mistral Large 2 all sit at 128K. Many enterprise applications, RAG pipelines, and agent frameworks are built around this tier - not because larger windows are unavailable, but because they are not always worth the cost or latency.

Tier 4 - under 32K tokens

Smaller and older models still fill specialized roles: classifiers, embedding models, fine-tuned task models, and edge-deployed local LLMs. Their 4K-32K windows are deliberate, optimizing for speed and inference cost over breadth.

Why Context Window Matters

Context window directly bounds what an LLM application can do in a single shot. The same model with a 8K window versus a 1M window is, functionally, two very different products:

• Document processing - A 128K window can hold a contract of a few hundred pages; an 8K window cannot. Whole-document analysis, cross-document comparison, and summarization at scale are gated by window size.
• Codebase reasoning - Large windows let coding assistants understand multi-file dependencies. A 1M-token window holds a meaningful slice of a real production codebase.
• Conversation depth - Multi-turn chatbots replay history into the prompt. A small window forces aggressive truncation; a large one preserves nuance and earlier instructions.
• Agent reliability - Tool-using AI agents accumulate observations from each step. Bigger windows let agents handle longer tasks before they hit the limit and lose state.
• RAG retrieval depth - A larger window allows retrieving more passages per question, increasing recall - though as discussed below, this is not a free win.

Context Window vs RAG vs Fine-Tuning

A common misconception is that larger context windows make retrieval-augmented generation (RAG) obsolete. In practice, the three approaches solve different problems and are usually combined.

• Context window is short-term, per-request memory. It is filled fresh every time and disappears after the response.
• RAG retrieves only the most relevant chunks of a much larger corpus and injects them into the window. It scales to terabytes of documents that no window - even 10M tokens - could hold.
• Fine-tuning bakes knowledge or behavior into the model weights themselves. The model learns patterns it then applies without needing the source content in the prompt.

Even with a 10M-token window, many production systems still rely on RAG for one simple reason: cost and latency scale with input size. Stuffing every query with millions of tokens is wasteful when only a few thousand are relevant. RAG keeps the prompt focused; the window provides the room to fit the retrieved snippets, the system prompt, and the question together.

The "Lost in the Middle" Problem

A model that advertises 1M tokens does not necessarily perform well at 1M tokens. Research by Liu et al. (2023) showed that LLMs perform best when the relevant information sits at the beginning or end of their context, with notable accuracy degradation for facts placed in the middle. The phenomenon is widely known as "lost in the middle."

In that study, GPT-3.5-Turbo answered questions over 20 retrieved documents. Accuracy was 75.8% when the document with the answer came first, 53.8% when it sat in the middle, and 63.2% when it came last, against 56.1% with no documents at all (closed-book). An answer buried in the middle did worse than no retrieval. The authors also found that extended-context models were not necessarily better at using their context than their shorter-window counterparts. Results vary by model and task, so measure your own.

Practical implications:

• Place critical instructions and key facts at the start or end of the prompt, not buried in the middle.
• Validate any application that relies on long-context recall with synthetic needle-in-a-haystack tests across the actual window length you intend to use.
• Treat advertised window size as a maximum, not a working capacity.

Cost and Performance Trade-offs

Context length is one of the dominant cost levers in LLM operations. Three considerations matter:

Cost

API providers charge per input and output token. At base rates, a query that uses 500K input tokens costs roughly four times as much as one that uses 128K, regardless of how much of that context the model actually uses to generate the answer. Some providers charge more above a threshold: OpenAI prices GPT-6.1 Sol prompts above 272K input tokens at 2x for input and cached input and 1.5x for output, for the full request.

Latency

Time to first token grows with input size. Long contexts can add several seconds - sometimes tens of seconds - of latency before the model starts streaming output. For interactive UIs, this is often the binding constraint, not cost.

Output budget

Some providers count input and output tokens against the same overall window. A long input can leave too little room for a long answer. Always check whether the provider exposes an explicit max_output_tokens parameter and whether output is counted separately.

Best Practices for Working With Context Windows

Across production systems, a small set of patterns consistently produce better results than naive context-stuffing.

• Retrieve, don't dump - use RAG to filter content down to what is actually relevant. Even with a 10M window, a focused 20K prompt often beats a 1M prompt for accuracy and cost. Anthropic's documentation notes that accuracy and recall degrade as token count grows, which it calls context rot.
• Place anchors at the edges - put the system prompt, critical instructions, and the user question near the start or end of the context. Reserve the middle for retrieved supporting content.
• Compress aggressively - summarize older conversation turns rather than replaying them verbatim.
• Chunk with overlap - when splitting long documents for retrieval, use overlapping chunks so important spans are not split across boundaries.
• Govern what enters the window - every token in context costs money and influences output. Filter retrieved content by ownership, classification, and lineage before injecting it, and by freshness if your platform tracks it. This is where active metadata earns its keep.
• Test at your real working size - never trust advertised limits. Run needle-in-a-haystack evals at the prompt size your application actually uses.
• Monitor token usage in production - instrument every call. Token consumption is the surest leading indicator of cost overruns and quality regressions.

The context window is a powerful resource, but it is not a substitute for grounded, governed access to enterprise data. The most reliable AI systems combine modest windows with strong retrieval over active metadata, data lineage, and a semantic layer - feeding the model exactly the context it needs, no more.

Sources

• Brown et al., Language Models are Few-Shot Learners (2020), arXiv:2005.14165
• Meta, Llama 4 model card
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023), arXiv:2307.03172
• Anthropic, Models overview and Context windows, Claude documentation
• OpenAI, Models and GPT-6.1 Sol, and Deprecations, OpenAI API documentation
• Google, Gemini models and Understand and count tokens, Gemini API documentation