Most tutorials that show you how to build a chatbot stop at the point where it works. You send a message, the model responds, you send another message, it responds again. Great. But those tutorials almost never talk about what happens to your token usage as the conversation gets longer, and that omission is where most people run into their first real production problem.
A chatbot that works fine in testing with five messages starts behaving differently at message twenty, costs three times what you expected at message thirty, and eventually hits a context limit that causes weird behavior you cannot explain until you understand what is actually happening under the hood.
This article is about building it right from the beginning so none of that happens to you.
Why conversation history is more expensive than it looks
Every time a user sends a message in a chatbot, you are not just sending that one message to the model. You are sending the entire conversation history, every previous message from the user and every previous response from the model, plus your system prompt, plus the new message. The model has no persistent memory between requests, so you have to give it the full context every single time.
This means the token cost of each message in a conversation is not fixed. It grows with every exchange. The first message might cost 600 tokens total. The second exchange costs 900 because now the first exchange is included as history. By the tenth exchange you might be sending 4,000 tokens per request even if the individual messages are short. By the twentieth exchange you might be at 8,000 or more.
If you built your cost model around the cost of a single request without accounting for this growth, your real costs in production will be significantly higher than your estimate, and they will keep growing as users have longer conversations.
The basic structure of a chatbot request
A well-structured chatbot request has three components that you manage separately. The system prompt, which stays constant and goes in the system parameter. The conversation history, which grows over time and goes in the messages array as alternating user and assistant messages. And the new user message, which gets appended to the history before each request.
The response from the model gets added to the history as an assistant message, so it becomes part of the context for the next request. That is the loop that makes the chatbot feel like it has memory, and it is also the loop that makes costs grow over time.
Keeping these three components conceptually separate from the beginning makes it much easier to manage them independently, which is important once you start thinking about optimization.
Setting a history limit from day one
The simplest way to prevent unbounded cost growth is to set a limit on how much history you pass with each request. Instead of passing every previous message, you pass only the most recent N exchanges, where N is a number you choose based on your context window budget and your cost targets.
A rolling window of the last ten exchanges works well for most conversational applications. Users rarely need the model to remember something from twenty messages ago, and the cost savings from limiting history are significant. The difference between passing five exchanges of history and twenty exchanges of history can easily be a factor of three or four in token cost per request.
The tradeoff is that the model loses access to earlier parts of the conversation. For some applications this matters and for others it does not. A customer support bot where each conversation is focused on a specific issue can usually work fine with a short history window. A research assistant where the user builds on earlier points throughout a long session might need a longer window.
According to this practical guide on LLM application architecture from The New Stack, managing conversation state and system prompts as separate concerns from the start is one of the most consistent patterns in well-architected AI applications, and one of the things most commonly retrofitted later by teams that did not plan for it upfront.
Summarization as an alternative to truncation
A rolling window that drops older messages entirely is simple but it loses information abruptly. A softer approach is to summarize older parts of the conversation rather than dropping them completely.
The way this works in practice is that once your history reaches a certain length, you take everything older than your rolling window and ask the model to summarize it into a compact paragraph or two. That summary replaces the raw history in your context, giving the model a compressed version of what was discussed earlier without taking up as much space as the full transcript.
This approach preserves more continuity than simple truncation and produces noticeably better behavior in long conversations where earlier context still matters. The cost is that you occasionally make an extra API call to generate the summary, but that cost is usually small compared to the savings from keeping the context compact.
The Context Window Visualizer on Prompt Toolbox is useful for understanding how much space different history management strategies leave you in practice, which helps you calibrate the right window size and summarization trigger point for your specific model and use case.
Keeping your system prompt tight
The system prompt is the one part of every request that never changes, which means every token in it gets paid for on every single request forever. A system prompt that is five hundred tokens longer than it needs to be costs you five hundred extra tokens on every message in every conversation your chatbot has.
Most system prompts that have been developed over time have accumulated more than they need. Instructions that were added to handle edge cases, clarifications that were added when something did not work right, caveats that seemed important but do not actually change the model's behavior. Reading your system prompt critically with cost in mind, and cutting everything that is not actively doing something, is one of the highest-return optimizations you can make before launch.
A good target for a chatbot system prompt is somewhere between two hundred and five hundred tokens for most applications. More than that is possible but it should be intentional, not accidental accumulation.
Handling the context limit gracefully
Even with good history management, conversations can eventually approach the context limit, especially if users paste in large amounts of text or ask about long documents. It is worth deciding upfront how your chatbot should handle this rather than letting the model's behavior degrade silently.
The options are to summarize and compress the history when you detect that you are approaching the limit, to start a fresh context with a summary of the old one, or to warn the user that the conversation is getting long and offer to start fresh. Which approach is right depends on your application, but having a deliberate strategy is better than letting the model quietly forget earlier parts of the conversation and confusing your users with inconsistent behavior.
Testing your chatbot with deliberately long conversations before you launch, pushing it to the point where history management kicks in, is one of the most useful things you can do to catch problems before real users do.
One thing most tutorials skip
Almost every chatbot tutorial focuses on the happy path where the user sends normal messages and the model responds helpfully. What they skip is what happens when the model's response is cut off because you hit the max tokens limit on the output side, which is a different limit from the context window.
If a user asks a question that requires a very long answer and you have set a low max tokens limit, the model will stop mid-sentence. That truncated response then gets added to the history as an assistant message, and the next request includes that incomplete response as context. This can cause confusing behavior that is hard to debug if you are not aware of it.
Setting your max tokens limit high enough to accommodate your longest expected responses, and handling the finish reason in the API response to detect when the model stopped because it hit the limit rather than because it finished naturally, are both worth building into your implementation from the start rather than discovering the problem in production.