Cutting AI running costs without cutting quality

The first AI invoice is usually a shock, and the reaction is usually the wrong one: switch to a cheaper model and hope nobody notices the quality drop.

In practice, most AI features are expensive because of how they were built, not because of what they cost per token. Fix the architecture and the same feature runs on a fraction of the spend with the same output. Here is where the money actually goes, in rough order of how much it is worth to you.

Where the money goes

The first surprise for most teams: input tokens usually dominate. Outputs are priced higher per token, but they are small — a few hundred tokens of answer. Meanwhile every single call re-sends the system prompt, the tool definitions, the retrieved documents and the entire conversation so far.

A chat feature that looks like "a few paragraphs in, a few paragraphs out" is often sending tens of thousands of tokens per turn and growing with every message. Retries and agent loops multiply it further: a ten-step agent that re-sends its full history each step pays for that history ten times.

1. Prompt caching — usually the biggest single win

If a substantial prefix of your prompt is identical between requests — and it almost always is, because system prompts and tool definitions do not change — you can cache it. Cached reads cost roughly a tenth of the normal input price, against a modest premium on the write. With a short-lived cache, you are ahead from the second request.

The catch, and it is the reason most teams see no benefit: caching is a prefix match. Any byte that changes anywhere in the prefix invalidates everything after it. The classic mistakes:

• Interpolating current date and time into the top of the system prompt. Every request is now unique.
• A request ID or session ID in the header of the prompt.
• Serialising a dictionary without sorting keys, so the byte order wobbles.
• Building the tool list per user, so no two users share a prefix.

The design rule follows directly: stable content first, volatile content last. Frozen system prompt and deterministic tool list at the front; the timestamp, the user's question, the fresh data at the end. Then verify — the API reports how many tokens were read from cache, and if that number is stubbornly zero, something in your prefix is changing.

2. Route by difficulty

Not every request needs your most capable model. A large share of real traffic is classification, extraction, formatting, simple lookups and short conversational turns — work a smaller, faster, cheaper model handles perfectly.

The pattern is a router: a small model handles the request by default; anything it flags as hard, or anything matching known-difficult criteria, escalates to the larger one. In most products the majority of traffic never needs to escalate, and the cost difference between model tiers is large enough that the split pays for itself immediately.

Two practical notes. Where the platform offers an effort or reasoning-depth control, that is a second dial on the same axis — often better than switching models outright, because you keep the same behaviour and just spend less thinking on easy work. And be aware that caches are model-specific: swapping models mid-conversation to save money throws away the cache and can cost you more than it saves.

3. Trim the context

Every token you retrieve, you pay for — on every subsequent turn it stays in the conversation. Vague retrieval is therefore expensive twice over: it costs more and it produces worse answers, because the signal is diluted.

The fixes are unglamorous and effective: retrieve fewer, better chunks by improving ranking rather than raising the limit; drop or summarise tool results once they have been used; trim conversation history rather than carrying every turn forever. Precision in retrieval is a quality lever and a cost lever at the same time — one of the few places where the two genuinely align.

4. Batch what is not urgent

If a workload does not need an immediate answer — overnight classification, bulk enrichment, generating summaries for a backlog — batch processing typically costs about half of real-time pricing for identical work. Nobody is waiting, so latency is free to trade away. This is the easiest saving on the list and the most commonly missed.

5. Measure per feature, not in aggregate

A single monthly number tells you nothing actionable. What you want is cost per request, broken down by feature, so you can see that the document summariser is fine and the "helpful" assistant on the settings page is quietly consuming most of the budget.

Track the same request over time, too. AI costs creep: a prompt grows, a context window fills, a retry loop gets sloppier. Without a per-feature baseline, nobody notices until the invoice arrives.

Do not optimise before you have evals

One warning, because we have watched it go wrong. Every lever above trades something. Route to a smaller model and some answers get worse. Trim context and you might cut the paragraph that mattered. Lower the reasoning effort and the hard cases start failing.

If you do not have an evaluation set — a fixed collection of real inputs with known-good outputs — you cannot tell the difference between a saving and a regression. Build the evals first, then optimise against them. Otherwise you are not reducing costs; you are quietly reducing quality and calling it efficiency.