What Is a Context Window: Why AI Forgets and How Tokens Work

Anyone asking what is a context window is asking how much text a model can hold in view at once. Anthropic defines it as all the text a language model can reference when generating a response, including the response itself, and calls it the model's working memory. Large language models predict one token at a time, so every prediction can draw only on what sits inside that window. Anything outside it stays invisible to the model.

What Is a Context Window Made Of: Prompts, Replies, Tool Results and Thinking

Anthropic counts more than the visible chat. Inside the window sit the system prompt, every message including tool results, images and documents, the tool definitions and the reply for the turn. Extended thinking counts too, billed as output tokens. A long answer therefore shrinks the room left for the next question.

Chat history is the part most people never see. A model keeps no memory between requests on its own, so apps resend earlier turns with each message, and building a first chatbot shows it: follow-up questions work only when the code passes a previous response ID forward and repeats the instructions.

Tokens Are the Unit: Roughly Three to Four English Characters Each

Tokens are chunks of text, not words. English averages roughly three to four characters per token, a rule of thumb that shifts with language and tokenizer, so exact counts need the provider's own counter, such as Anthropic's token counting API.

Images and documents count as well. Anthropic allows up to 600 images or PDF pages per request on its 1M-token models, and the voice AI explainer argues that longer context matters most in extended sessions, where it spares people from re-explaining themselves.

Billing runs on tokens too. GPT-6 Astra lists at $10 per million input tokens and $50 per million output tokens in the API, with fast mode doubling both, while Grok 4.7 charges $2 and $6. Plans meter tokens as well: starting in December, Plus, Pro and Business subscribers share one token-based usage allowance across ChatGPT and Codex, part of the OpenAI platform.

Why AI Forgets: Overflow, Context Rot and the Lost-in-the-Middle Gap

Forgetting comes in two forms, a hard limit and a soft one. Both matter. Hard limits act like a wall: if input alone exceeds the window, the Anthropic API returns a 400 error reading "prompt is too long", and on Claude 4.5 models and newer, a request that leaves too little room for the reply stops mid-generation with the stop reason model_context_window_exceeded.

Products usually hide the wall by summarizing. Anthropic calls server-side compaction its primary strategy for long-running conversations, and the agent harness explainer describes the same move at the runtime layer, where older turns get summarized and a compact task state replaces raw history. Summaries lose detail. Kimi K3 shows the trade-off: moving from its 1M model to the 256K one while context exceeds 256K may trigger compaction.

Soft limits arrive quietly. Anthropic warns that as token count grows, accuracy and recall degrade, a phenomenon it calls context rot. Chroma's July 14, 2025 report tested 18 models, including Claude Opus 4, GPT-4.1 and Gemini 2.5 Pro, and found that models do not use their context uniformly, with reliability declining as input grows even on simple tasks.

Position matters as well. Nelson Liu and six co-authors found in the 2023 "Lost in the Middle" paper that performance is often highest when relevant information sits at the beginning or end of the input, and drops noticeably when it sits in the middle, even for models built for long contexts. A model that misses a buried fact still answers fluently, since hallucinations come from predicting plausible text rather than looking facts up.

Window size and recall are separate specs. Meta's Muse Wonder models accept 1,048,576 tokens, and as the Muse Wonder benchmark comparison points out, accepting lots of input differs from reasoning over it reliably. A bigger window works more like a longer desk than a better memory.

Context Window Sizes: 1M Is Common, While Kimi, Grok and Others Sit Lower

Anthropic lists 1M-token windows for Claude Fable 5.2, Opus 5.5, Sonnet 5.5 and Haiku 5.5, among others, at standard pricing, except that Haiku 5.5 prompts over 100,000 tokens cost more. Other vendors differ by model and plan.

Raw size misleads for code. A 1M window does not guarantee a whole repository fits, since generated files, logs and tool output also consume context, as the Kimi K3 context window explainer cautions.

Mid-sized windows persist because size costs money. Amazon Bedrock added Grok 4.6 with a 500K-token window on Aug. 19, and Kimi charges about twice the quota for its 1M configuration compared with 256K.

Agents Fill a Window Fastest: Tool Results, Logs, Pages and Persistent Memory

Agents tend to fill a window faster than chat. Agentic AI plans steps, carries them out, checks progress and adjusts after failures, and each pass adds tool results and logs to the history. An AI agent differs from a chatbot in exactly that loop, and the agent versus chatbot breakdown notes that agents choose each next step after seeing earlier results.

Tool definitions count before the first question. Anthropic counts them toward the window, and MCP clients discover a server's tools automatically after connecting, so each added server can add text to every request. Tool results count too, and pages read through tools such as Claude Code's built-in browser appear to follow the same rule.

Persistent agents lean on stored memory. OpenAI Dots run on GPT-6 Astra with their own cloud computers, and the Meta Muse, Grok Bot and Dots comparison describes Dots drawing on saved memories and linked app data to carry work across conversations.

Platform choices change the picture. Claude absorbed Cowork into regular chat, so larger tasks that continue after a laptop closes start from any conversation, per the Cowork merge, while Vercel's Eve treats models as swappable parts, which ties window size to whichever model sits behind the agent. Adding only the tools a job needs, as the guide to creating an AI agent advises, also keeps fewer definitions in the window.

Prompt Caching Cuts the Price of Repeated Context, Not the Token Count

Cached prefixes still occupy the window. Anthropic says caching changes the price of those tokens, not whether they count. Cache writes cost 1.25 times the base input price for the default 5-minute lifetime and 2 times for a 1-hour lifetime.

Each use refreshes the lifetime for free. Discounts like these feed straight into AI pricing comparisons, where a model with a higher list rate can cost less across long, repetitive agent sessions.

Five Habits That Stretch a Window Without Paying for a Bigger One

1. Start a fresh chat when the topic changes, since every earlier message keeps occupying the window and adding noise.
2. Put the instruction and key facts at the start or end of a long prompt, where the Lost in the Middle paper found recall strongest.
3. Paste only the relevant section of a document, or retrieve it, instead of dumping a whole file.
4. Restate constraints in long sessions, and resend instructions with every request in API-built tools.
5. Keep passwords, API keys and access tokens out of prompts and swap names and account numbers for placeholders, as the guide to protecting data in AI chatbots advises.

Task budgets remain in beta. Anthropic points Opus 4.7 and later, Sonnet 5.5, Haiku 5.5 and the Fable and Mythos 5.x models to them instead of the automatic token-budget tags that Sonnet 5 and Sonnet 4.6 still receive. Server-side compaction is still in beta for Claude 4.6 and later models, so how products manage a full window keeps changing.

Related Articles

xAI Launches Grok 4.6 at $2 per Million Tokens - Half the Price of Rival Frontier ModelsClaude Opus 5 Launches: Same Price as Opus 4.8, Within 0.5% of Fable 5 on Coding BenchmarksClaude Opus 5 vs Fable 5: Six Benchmarks, Side-by-Side Pricing, and When to Switch