AI Explained
Context window: the model's working memory
Why does a long AI chat forget the beginning? Meet the context window: everything a model can see at once, measured in tokens, and why bigger isn't always better.
On this page
Why does a long AI chat seem to forget how it started? Because every model has a limit on how much it can see at once. That limit is the context window.
The model's working memory
Anthropic describes the context window as "all the text a language model can reference when generating a response, including the response itself", a working memory for the model. It's measured in tokens, the chunks of text models read.
The model has no other memory of your conversation. Each time it answers, it re-reads whatever is in the window. If something isn't in there, the model doesn't know it.
Everything shares the same space
A lot more goes into the window than your latest message:
- the system prompt and instructions,
- the whole chat history so far,
- files and documents you attached,
- tool definitions and tool results,
- and the reply the model is writing right now.
All of it competes for the same budget. A big PDF or a long tool result can use up much of the space by itself.
When it fills up
Something has to give. Apps usually do one of two things:
- Cut the oldest messages, which is why early details vanish.
- Summarize them. Anthropic calls this compaction: it "replaces the older turns of a conversation with a summary" so a long session can keep going.
Bigger isn't always better
Windows have grown fast, and many current models take around a million tokens. But more context isn't automatically better. As the window fills, accuracy and recall can drop, a problem often called context rot. Important details get lost in the middle of a huge prompt.
How to use it well
- Put only what the task needs in the prompt.
- Start a fresh chat when you change topics.
- Summarize long threads before continuing.
- Put the most important instructions where they're easy to find, near the start or the end.
Keep it lean, keep it sharp.