You built a chatbot. It works great in testing. The first few messages are sharp, helpful, exactly what you wanted. And then somewhere around message ten or twelve something starts to feel off. The model contradicts something it said earlier. It forgets a detail the user mentioned three exchanges ago. It starts giving answers that feel generic and disconnected from the thread of the conversation.
You go back and check the model. The model is fine. You check your API calls. Everything is going through. You check your logs. No errors. So you do what most people do in this situation: you assume it is just a model limitation, you add a note in your mental backlog to look into it later, and you ship it anyway.
This is a mistake. And it is an extremely common one.
What you are experiencing has a name. Researchers call it context rot. And it is not a model problem. It is an architecture problem. Specifically, it is your architecture problem, and it is almost always fixable.
What context rot actually is
Context rot is what happens when a conversation gets long enough that the model starts losing meaningful access to earlier parts of it. The model does not hard-delete old information the moment it falls outside the context window. What actually happens is more insidious than that.
As conversations grow longer, older content gets pushed further and further back in the context. Research has consistently shown that language models pay less attention to information in the middle of long contexts than to information at the beginning or the end. So even before you hit the hard context limit, the model is already treating your early messages as background noise rather than relevant information. It is technically seeing that content but not really processing it with the same weight it gives to recent content.
Then when you actually hit the context limit, the oldest messages start getting dropped entirely. The model does not tell you this happened. It does not say "by the way I no longer remember what the user said in message two." It just quietly loses that information and keeps responding with the same confident tone it always uses, filling in the gaps with whatever seems most plausible based on what it can still see.
As UX Collective noted in their piece on the forgotten conversation problem in AI chat, the model will often act like it remembers things it no longer has access to, because it is trained to be helpful and maintain conversational flow, which makes the degradation worse because users cannot tell the difference between the model actually remembering something and the model confidently making up something plausible.
That is the part that should concern you as a builder. Your users are not getting error messages. They are getting wrong answers delivered with confidence. That is worse.
Why this is your problem and not the model's problem
Here is the uncomfortable truth. The model is doing exactly what it was designed to do. It is processing whatever context you give it and generating a response based on that context. If the context you are giving it is incomplete, poorly managed, or inefficiently structured, you are going to get incomplete, poorly structured outputs. The model is not responsible for managing its own context. You are.
Most chatbot implementations do one of two things with conversation history. Either they pass the full history every time and hope it never gets too long, or they truncate it at some arbitrary message count without thinking about what gets lost when they do. Neither of these is a real strategy. Both of them produce the same user experience: a chatbot that works well in short conversations and quietly falls apart in long ones.
A real strategy means making deliberate decisions about what information needs to persist across the entire conversation, what information is only relevant for the last few exchanges, and how to handle the transition when a conversation gets long enough that you can no longer keep everything.
What you should actually be doing
The most effective approach for most chatbot applications is a hybrid that combines a rolling window of recent exchanges with a compact summary of older content.
The rolling window handles recency. The last five to eight exchanges stay in the context verbatim because recent context matters most for the immediate back-and-forth of a conversation. The model needs to know what was just said to give a coherent response to what is being said now.
The summary handles continuity. Instead of keeping every earlier message verbatim, you periodically take the oldest messages in your window and ask the model to compress them into a concise summary of what was discussed, what was decided, and what details matter. That summary takes up far fewer tokens than the original messages while preserving the information that actually needs to persist.
The Context Window Visualizer on Prompt Toolbox is useful here because it lets you see how quickly different conversation structures fill the context window of the model you are using, which helps you figure out at what point in a conversation you need your summarization to kick in.
The thing nobody builds until users complain about it
The dirty secret of most AI chatbot products is that context management is almost always an afterthought. It gets built as a quick fix after users start complaining that the bot seems confused, which is the worst possible time to think about it because now you are retrofitting it into an architecture that was not designed for it.
The teams that handle this well build it from day one. They decide upfront what their maximum context size will be, how they will handle conversations that exceed it, and what information is critical enough to always be in the context regardless of conversation length. Those decisions are not hard to make. They just require making them before you ship instead of after.
If you have already shipped and you are now dealing with user complaints about a chatbot that seems to get worse over time, the fix is the same. It just requires a little more work to retrofit. Add context management, implement summarization for long conversations, and test specifically with long conversation scenarios rather than only with the happy-path short conversations that everything always looks fine on.
One more thing
There is a specific failure mode that is worth calling out separately because it is more insidious than general context rot. If your chatbot application injects any kind of dynamic context into each request, user profile data, relevant documents, search results, whatever your application pulls in to help the model answer, that injected content is taking up context window space that used to be available for conversation history.
As your injected context grows, your effective conversation window shrinks. A chatbot that was handling twenty message conversations fine starts struggling at twelve messages because the injected context expanded and pushed out the conversation history faster than before. This is invisible unless you are actively monitoring how your context budget is being allocated across the different components of each request.
Instrument your application to log context usage broken down by component. Know how many tokens your system prompt uses, how many your injected context uses, how many your conversation history uses, and how many are left for the model to respond in. That visibility is what lets you catch context rot before your users do instead of after.