Every few months someone writes a take about how context windows are still not big enough, how AI models still cannot handle long conversations, how the technology has a fundamental limitation that nobody has figured out how to solve yet. These takes are almost always wrong about the cause and therefore useless about the solution.
Context windows are bigger than they have ever been. Some models support a million tokens. That is roughly seven hundred and fifty thousand words. That is longer than War and Peace. Twice.
And yet people are still running into problems with long conversations. Models still forget things. Outputs still degrade. Users still notice that the AI seems to lose the thread somewhere around message fifteen.
The context window did not cause this. You did. Specifically, you caused it by treating conversation history as something that just happens rather than something you design.
What lazy history management actually looks like
Here is the most common pattern in production chatbot applications. Every user message gets appended to a history array. Every model response gets appended to the same array. That array gets passed with every request. Nothing ever gets removed from the array. Nothing ever gets summarized. Nothing ever gets prioritized.
This approach has a name. It is called hoping for the best. It works fine until it does not, and when it stops working it stops working in ways that are hard to diagnose because nothing throws an error. The model just starts giving worse answers and users start losing trust and you spend two days trying to figure out if something changed in the model before someone finally looks at the actual token counts being sent per request and has a moment of painful clarity.
The thing is, this pattern is not even a real implementation. It is the absence of an implementation. Real history management is a deliberate set of decisions about what stays in context, what gets compressed, what gets dropped, and when each of those things happens. Passing a raw array and calling it done is just deferring those decisions to the moment when the lack of them causes a problem.
The context rot research that should scare you
Kelly Hong from Chroma coined the term context rot to describe something that researchers have been finding consistently across model evaluations. As Hamel Husain detailed in his breakdown of her research on context rot, the finding is that LLM performance degrades with longer inputs in ways that are not uniform and not predictable. Models do not process a million token context with the same reliability they process a ten thousand token context. Performance drops, and it drops in specific patterns depending on where in the context the relevant information sits.
The practically important finding is that information in the middle of a very long context gets less attention than information at the beginning or the end. This means that a conversation where the user told you something important in message three, and that message is now buried in the middle of forty exchanges of history, is a conversation where the model may not reliably use that information even though it is technically within the context window.
More context is not always better. Sometimes it is worse. The model you are asking to process everything is not actually processing everything equally. Knowing that changes how you should think about what to put in the context and in what order.
What deliberate history management actually looks like
Will Larson described the problem and a practical solution in his post on context window compaction for AI agents, noting that long-running workflows inevitably run out of context space and that the architecture needs to account for compaction from the start rather than treating it as an edge case.
The pattern that works is not complicated but it requires making three decisions upfront that most developers skip.
The first decision is what your maximum history window is. Not the maximum context window of the model, your maximum history window. The amount of raw conversation history you are willing to pass before you start compressing. This should be significantly smaller than the context limit, because you need to leave room for your system prompt, injected context, and the model's response, and because of the context rot problem, keeping history shorter and denser is often better than keeping it longer and raw.
The second decision is what compression looks like when you hit that window. The most effective approach for most applications is to take everything older than your rolling window, pass it to the model with an instruction to summarize it into a compact paragraph capturing the key points and decisions, and replace the raw history with that summary. The summary takes up a fraction of the tokens while preserving the thread of the conversation for the model to reference.
The third decision is what information is important enough to pin, meaning it always stays in context regardless of how long the conversation gets. If the user told you their name, their role, and their specific problem in message one, that information might need to stay explicitly accessible even when message one is no longer in the rolling window. A pinned context section that carries forward only the most critical information from older parts of the conversation is often more reliable than hoping the model extracts it from a summary.
The failure mode nobody talks about
There is a specific way history management goes wrong that is more insidious than running out of context. It is when the history contains bad information.
If the model gave an incorrect answer in message five, and you have been passing that incorrect answer as part of the conversation history ever since, the model is now operating in a context where the incorrect answer is presented as established fact. It will not necessarily correct it. It might build on it. And debugging why your chatbot is consistently wrong about a specific thing requires going back through the history to find the moment the error entered the context and everything downstream of it.
This is the argument for not just managing the size of your history but occasionally reviewing its quality. Summaries that get generated programmatically should capture the correct information, not just the most recent information. In applications where accuracy matters, having a mechanism to correct or remove bad information from the history rather than just compressing it is worth the engineering effort.
The practical thing you can do today
If you have a chatbot in production with no history management, the first step is to add instrumentation that shows you the token count breakdown per request, specifically how many tokens are coming from conversation history versus everything else.
Look at that number for a long conversation versus a short one. See how fast it grows. Calculate what it looks like at conversation length twenty versus conversation length five. Then decide whether what you see is acceptable or whether you need to build something.
The Context Window Visualizer on Prompt Toolbox shows you how different models handle context size, which gives you a useful reference point when you are deciding what your maximum history window should be relative to the model you are using.
The context window is not your enemy. It is a tool with known characteristics that you can design around. Designing around it is not optional if you want your application to behave reliably at conversation lengths that real users actually reach. It is just engineering.
