At some point in almost every AI project, the same conversation happens. The costs are higher than expected, someone suggests cutting the quality of the model or the responses to bring them down, and then the debate starts about whether that tradeoff is worth it. The thing is, that's usually not the right conversation to be having. In most cases there's a significant amount of token waste that can be cut without touching quality at all, and finding that waste is almost always the better first step.
This article is about where that waste tends to hide and how to get rid of it.
The difference between necessary tokens and wasted tokens
Not all tokens in a prompt are doing the same amount of work. Some are essential, they carry information the model needs to understand your task, follow your instructions, or generate a useful response. Others are redundant, they repeat information that's already clear from context, explain things the model already knows, or add caveats and qualifications that don't actually change what the model does.
The goal of cost optimization is not to make your prompts as short as possible, it's to cut the tokens that aren't pulling their weight while keeping the ones that are. A prompt that's half the length but produces worse results isn't an optimization, it's a tradeoff. A prompt that's half the length and produces the same results is a genuine win.
The first step is to get a clear picture of where your tokens are actually going. Paste your full prompt, including any examples, context, and instructions, into the Tokens per Dollar Calculator and look at the breakdown. Most people are surprised by how many tokens their system prompt alone is using before the user's actual input even arrives.
The system prompt is usually the first place to look
System prompts tend to grow over time. You add an instruction to handle an edge case, then another, then a clarification for an instruction that wasn't working, then a few more caveats, and before long you have a system prompt that's three times longer than it needs to be with several instructions that are either redundant or no longer relevant.
Reading your system prompt critically with cost in mind is different from reading it for correctness. When you're checking for correctness, you're asking "does this cover what I need?" When you're checking for efficiency, you're asking "is every sentence here doing something the model wouldn't do without it?"
A lot of system prompts contain instructions like "always be helpful and polite" or "make sure your responses are accurate." These sound reasonable but they don't actually change the model's behavior because the model already behaves that way by default. Cutting them doesn't change your outputs but it does save tokens on every single request.
According to VentureBeat's reporting on enterprise AI cost management, system prompt optimization is consistently one of the highest-return activities for teams trying to reduce AI API costs, because the savings compound across every request rather than applying to just one.
Conversation history is the other big one
If your application maintains conversation history, that history grows with every exchange and gets sent with every subsequent request. By the time a conversation is ten or fifteen exchanges long, a substantial portion of your token budget is being used just to give the model context it mostly already has.
The most effective way to manage this is to summarize older parts of the conversation rather than keeping them verbatim. Instead of passing the full text of every previous message, you periodically condense the earlier part of the conversation into a short summary and pass that instead. The model loses some fine detail but keeps the important thread, and your token count stays manageable even in long conversations.
Another approach is to set a rolling window, only passing the last several exchanges rather than the full history. This works well for many use cases where recent context matters more than what was said at the start of the conversation.
On the output side, specificity beats brevity instructions
One of the most common pieces of advice for reducing output tokens is to tell the model to "be concise" or "keep it short." This advice isn't wrong but it's not very effective either, because those instructions are too vague for the model to act on consistently.
What works better is being specific about the format and length you want. "Answer in three sentences" is much more effective than "be concise." "Give me a bulleted list with no more than five items" is more effective than "keep it brief." The model follows specific format instructions reliably, and specific instructions about length produce more consistent results than general ones.
Asking for structured output instead of prose is another option that often reduces token counts naturally. A JSON response or a simple list tends to use fewer tokens than the same information written in paragraph form, because prose includes transitional language and connective tissue that structured formats skip.
Testing before and after
The only way to know whether your optimizations are working is to measure them. Run a realistic sample of inputs through your optimized prompt, compare the token counts to your baseline, and look at the output quality side by side. Cost optimization that makes outputs noticeably worse isn't a real optimization, it's just cutting corners.
The good news is that in most cases, the first round of optimization finds meaningful savings without any quality loss at all. The redundant instructions, the verbose system prompt sections, the verbatim history, these are all things you can cut without the model noticing. The harder tradeoffs only come later, and by that point you've usually already found enough savings to make the project work.