There is a particular kind of cost problem that shows up in AI applications once they have been live for a while. It is not a spike that happens overnight. It is a slow drift where costs creep up month after month, not dramatically enough to trigger an alarm but consistently enough that six months later your bill is twice what it was at launch and nobody can clearly explain why.
This happens because AI API costs are driven by a handful of variables that change over time in ways that are easy to miss if you are not actively watching them. Conversations get longer as users become more engaged. System prompts accumulate new instructions. New features add token-heavy context to requests. None of these feel significant individually but they compound.
This article is about how to find where your tokens are actually going and what to do about it.
Start with a usage breakdown, not a total number
The first mistake most teams make when trying to reduce AI costs is looking at the total monthly spend and trying to bring that number down without understanding what is driving it. That approach leads to blunt cuts that often hurt quality without addressing the real sources of waste.
A more useful starting point is to break down your token usage by request type. Different features or workflows in your application probably have very different token profiles. A simple classification request might use two hundred tokens. A document summarization request might use three thousand. A conversational session might use anywhere from five hundred to five thousand tokens depending on how long the conversation runs.
If you can instrument your application to log token counts by request type, you will almost always find that a small number of request types account for a large majority of your total token spend. Those are the ones worth focusing on first, not the low-volume low-cost requests that are easy targets but move the needle very little.
The system prompt audit
System prompts are one of the highest-leverage places to look for savings because every token you cut from a system prompt saves you that many tokens on every single request your application processes, forever. A system prompt that has grown to two thousand tokens over time and could realistically be eight hundred tokens is costing you twelve hundred tokens per request more than necessary.
The way to audit a system prompt is to read every sentence and ask whether the model's behavior would actually change if you removed it. Instructions that repeat things the model does by default, caveats that do not affect output, explanations that were added for human readers rather than for the model, these are all candidates for removal.
A useful test is to run a set of representative requests with your full system prompt and then again with a trimmed version, comparing the outputs side by side. In most cases you will find that a significant portion of system prompt content can be removed with no meaningful change in output quality.
Conversation history is usually the biggest driver of cost growth
If your application maintains conversation history, that history is almost certainly the main reason your costs grow over time. Every exchange adds tokens to every subsequent request in that conversation, and as your users have longer and more engaged conversations, the average token cost per request goes up even if nothing else changes.
The audit here is to look at the distribution of conversation lengths in your application. What is the average number of exchanges per session? What does the long tail look like? If you have users having thirty or forty exchange conversations, those sessions are costing dramatically more per request than your average case, and they may be pulling your average up significantly.
As TechCrunch noted in their coverage of hidden AI costs that can derail innovation budgets, variable costs in AI applications tend to grow in ways that are invisible until they are already a serious problem, precisely because they are driven by usage patterns that change gradually rather than by a single discrete decision.
Output token variance is harder to see but worth measuring
Input tokens are relatively predictable once you know your system prompt size and your history management strategy. Output tokens are harder to control because the model decides how many to use, and that decision varies a lot based on what users ask.
The audit here is to look at the distribution of output token counts across your requests. What is the average? What is the 90th percentile? What is the maximum you have seen? If your average output is three hundred tokens but your 90th percentile is twelve hundred tokens, you have significant variance that is probably driven by a specific type of user request that triggers long responses.
Once you identify what types of requests are generating your longest outputs, you can decide whether to add more specific length instructions for those cases, restructure how those requests are handled, or accept the cost as justified by the value those responses provide.
Caching is underused by most teams
Most major AI providers offer some form of prompt caching where repeated content in your requests is charged at a reduced rate. Anthropic offers prompt caching that significantly reduces the cost of repeated system prompt content. OpenAI has a similar feature for longer prompts. If your application sends the same system prompt or the same large block of context with every request and you are not using caching, you are paying full price for tokens that the provider has already processed.
Setting up caching is usually a small implementation change for a meaningful cost reduction, especially for applications with long system prompts or applications that inject the same reference material into many requests. It is one of the first things worth checking if you have not already done it.
Model routing is worth considering at scale
Not every request your application handles needs the most capable model you have access to. Simple classification tasks, basic extraction, short question and answer interactions, these can often be handled well by a smaller cheaper model. More complex reasoning, nuanced writing, or tasks that require deep understanding benefit from a larger model.
If your application handles a mix of request types, routing simpler requests to a cheaper model while keeping complex ones on your primary model can reduce costs significantly without users noticing any difference in quality for the tasks where the cheaper model performs just as well.
The implementation requires some upfront work to classify requests and evaluate model performance on each category, but the ongoing savings can be substantial at scale. A rough rule of thumb is that if thirty to forty percent of your requests are genuinely simple tasks, routing those to a cheaper model can reduce your total spend by fifteen to twenty percent without touching the requests where quality matters most.
The ongoing monitoring that prevents the problem from coming back
Auditing your token usage once is useful. Having ongoing visibility into it is what prevents costs from drifting up again after you have addressed the initial issues.
The practical minimum is to track average tokens per request by request type over time, with an alert if any category rises significantly above its baseline. That kind of monitoring catches problems like a system prompt that got longer after a deployment, a new feature that adds more context than expected, or a change in user behavior that is driving longer conversations, before they compound into a meaningful cost increase.
The Tokens per Dollar Calculator on Prompt Toolbox is useful for translating your token metrics into dollar terms quickly, which makes it easier to communicate the impact of optimizations to stakeholders who think in costs rather than token counts.