There is a specific kind of panic that happens when you check your API billing dashboard for the first time after a real launch. The number is higher than you expected, sometimes by a lot, and the worst part is that you are not entirely sure why. You did the math beforehand. You ran the numbers. And somehow the actual bill does not match what you calculated.
This happens to almost everyone building on AI APIs for the first time, and it happens for the same reasons over and over. This article is about those reasons and how to build a cost estimate that actually holds up when real users start using your product.
Why the estimates people make before launch are usually wrong
The most common reason estimates are wrong is that they are based on a single happy-path request rather than the full range of requests your application actually handles. You take one example prompt, count the tokens, multiply by your expected request volume, and call it a cost estimate. But that one example is usually the cleanest, shortest version of a request your application will ever see.
Real users write longer messages than you expect. Real conversations go longer than your test conversations did. Real use cases surface edge cases where your application sends more context than the average request. All of those things push your actual token counts higher than your example-based estimate, and none of them show up until you are in production.
The second reason is that most people estimate input tokens carefully and guess at output tokens. Output tokens are harder to estimate because the model decides how many it uses, and models tend to write longer responses than you expect when users ask open-ended questions. Since output tokens are priced higher than input tokens on most models, underestimating them means underestimating the expensive half of your costs.
Start with a realistic sample, not an ideal one
The foundation of a good cost estimate is measuring token counts on realistic inputs, not carefully constructed examples. That means taking actual user messages if you have them from a beta or early test, or constructing prompts that represent the full range of what users might send including the longer and more complex ones, not just the simple cases.
Run those realistic inputs through your full prompt stack, meaning the system prompt, any context you inject, the conversation history if your app maintains one, and the user message, and count the total tokens for each one. Then look at the distribution. What is the average? What does a longer-than-average request look like? What is a realistic worst case?
The Tokens per Dollar Calculator on Prompt Toolbox makes it easy to take those token counts and see what they translate to in dollar terms across different models, which gives you a cost-per-request number that reflects your actual usage pattern rather than an idealized one.
The conversation history multiplier
If your application maintains conversation history and passes it with every request, your cost per request is not a fixed number. It grows with every exchange in a conversation because every new request includes everything that was said before it.
The first message in a conversation might cost 800 tokens total. The second exchange might cost 1,400 tokens because the first exchange is now part of the context. By the fifth exchange you might be at 3,500 tokens per request even if the individual messages are short. If your cost estimate assumed a fixed cost per request and ignored the conversation history growth, it could be off by a factor of three or four for users who have long conversations.
The way to account for this is to estimate an average conversation length in exchanges for your application, calculate the token cost at each step of a typical conversation including the growing history, and average those costs across the conversation. That gives you a cost per conversation rather than a cost per request, which is often a more useful unit for planning.
Output tokens are harder to control than you think
One of the things that surprises developers most is how much output token counts vary depending on what users ask. If your application lets users ask open-ended questions, some of those questions will generate responses that are three or four times longer than your average case. And since you pay for those output tokens either way, a small number of unexpectedly long responses can meaningfully affect your monthly total.
There are things you can do to reduce output token variance. Giving the model explicit length instructions in your system prompt, asking for structured responses instead of open-ended prose, and setting a max tokens limit on your API calls all help keep output lengths more predictable. But none of them eliminate variance entirely, and your cost estimate should assume some spread around your average rather than assuming every response will be exactly average length.
According to Wired's reporting on the economics of AI products, output token variance is one of the most commonly underestimated cost factors in consumer AI applications, particularly in products where users have significant latitude in how they interact with the model.
Building a cost model that accounts for scale
Once you have a realistic cost per conversation, the next step is to think about how that scales with your user base. This sounds simple but there are a few things that make it more complicated than just multiplying conversations by cost per conversation.
First, usage is almost never uniform. You will have power users who have many long conversations and light users who have one short interaction and never come back. The distribution of usage across your users matters a lot for your total cost, and the shape of that distribution is hard to predict before launch. Building some conservatism into your estimate to account for the possibility that your active users are more engaged than average is usually the right call.
Second, think about which requests in your application are AI-powered and which are not. Some features might trigger multiple AI calls per user action, for example if you do a retrieval step and then a generation step separately. Each of those is a billable request, and the cost model needs to account for all of them not just the ones that are obvious.
Third, check whether your provider offers any pricing features that could reduce your costs at scale. Prompt caching, where the API charges less for tokens that were part of a previous request, can significantly reduce costs for applications with long system prompts or repeated context. Batch processing APIs, where you send requests in bulk at a discount, can also help if your use case can tolerate asynchronous processing.
The number you want before you launch
Before you launch anything with real users, you want three numbers: your expected cost per active user per month in your base case, the same number in a scenario where usage is higher than expected, and a rough sense of at what user count your costs become a business problem if your monetization does not cover them.
Those three numbers give you a clear picture of your financial risk and what levers you have to manage it. They are not hard to calculate once you have realistic token counts and a clear model of how your application works, and having them before launch is infinitely better than trying to figure out what went wrong after your first real bill arrives.
