LLM API Cost Optimization (2026): Caching, Batching & Routing

Quick answer

The three levers that actually move an LLM API bill are batching (a published 50% discount at every major provider), prompt caching (cache reads at roughly one tenth of base input price), and routing each task to the cheapest model tier that passes its quality bar. Everything else is rounding error by comparison. The catch is that each lever now carries a condition - batch has a latency cost, caching has a write cost, and routing has a quality cost - and a fourth category of pricing modifiers quietly erases the savings if nobody is watching. The verified 2026 numbers and the break-even maths are below.

We already publish the single-provider version of this playbook in OpenAI API cost optimization. This one is cross-provider, and it is built entirely on prices we pulled from vendor pricing pages in September 2026 rather than from memory. That distinction matters more than it sounds: LLM pricing moved several times in 2026, and at least one provider has a dated increase already on the calendar.

What a token actually costs in September 2026

Here is the shape of the market at the tiers most production workloads live on. All figures are USD per million tokens, standard synchronous tier, checked 6 September 2026.

Sources: OpenAI pricing, Anthropic pricing, Gemini API pricing.

Two things jump out. First, the spread between the top and bottom row is roughly 200x on input and 125x on output. That is the business case for model routing in one number. Second, cached input is consistently about a tenth of base input across all three vendors. Caching is not a micro-optimization; it is a first-class pricing tier.

One dated item belongs in your 2027 forecast now: Google’s pricing page shows Gemini 3.8, 3.7, and 3.6 Flash roughly doubling on 1 January 2027, from $0.75 to $1.50 input and $3.75 to $7.50 output. If your unit economics are built on 2026 Flash rates, they have an expiry date.

Lever 1: batching, the discount you get for being patient

Every major provider will halve your bill for work that does not need an answer this second. Anthropic publishes the batch table outright - Opus 5 drops from $5/$25 to $2.50/$12.50, Sonnet 5 from $2/$10 to $1/$5, Haiku 4.5 from $1/$5 to $0.50/$2.50 - and describes it as “a 50% discount on both input and output tokens”. Google’s pricing page lists batch rates at exactly half standard for every model. OpenAI documents the same 50% discount in its Batch API guide, with batches completing within 24 hours.

Teams skip this lever because it feels unglamorous, and it is almost always the largest one available. Go through your workloads and ask one question of each: does a human wait on this response? Nightly enrichment, backfills, evaluation suites, document classification, summarization pipelines, and embedding refreshes all answer no. Those workloads are paying a 100% premium for latency nobody consumes.

Plan for the constraint: turnaround is a target rather than an SLA, and Anthropic caps in-queue batch requests by tier. Batch also does not apply everywhere - the discount does not extend to Managed Agents sessions or stack with Fast mode, because both are interactive by definition.

Lever 2: prompt caching, which is no longer free

Caching used to be a pure win. In 2026 it is a trade with a break-even point, and the teams claiming savings they never earned are the ones who never checked the hit rate.

Anthropic publishes the whole mechanism, which makes it the clearest worked example available:

Their guidance is unambiguous: caching “pays off after one cache read for the 5-minute duration (1.25x write), or after two cache reads for the 1-hour duration (2x write)”. So the governing metric is not “is caching enabled” but cache hit rate per breakpoint. A long system prompt cached on a route that fires twice an hour with a 5-minute TTL is a cost increase wearing the costume of an optimization.

Three rules fall out of that:

• Cache the stable prefix, not the request. System prompts, tool definitions, few-shot blocks, and retrieved documents reused across calls are cacheable. Per-user context appended at the end is not.
• Match TTL to traffic shape. Pick the 1-hour write only where you can demonstrate two or more reads inside the hour. Otherwise the 5-minute write is cheaper.
• Instrument the hit rate. Anthropic returns cache_read_input_tokens and cache_creation_input_tokens in the usage block on every response. Log both. If you are not logging them, you do not know whether caching is saving or costing you money.

There is a throughput bonus that rarely makes it into cost models: on most Claude models, cache-read tokens do not count toward input-tokens-per-minute rate limits. Caching buys headroom as well as discount.

Lever 3: routing, where the 200x lives

Routing has the biggest ceiling and the most friction, because it requires an opinion about quality. The mechanics get simple once you accept a boring premise: most production LLM calls are not reasoning tasks. They are classification, extraction, reformatting, and short summarization. Those are mini-tier and nano-tier jobs.

Build the routing policy as a table of task classes rather than a per-feature decision:

Then enforce it. A routing policy that lives in a Notion page is a suggestion; a routing policy expressed as a default in your gateway, with an exception that requires justification, is governance. This is the same shift-left principle we apply to infrastructure cost in shift-left cost management - the cheapest place to catch an expensive route is the pull request that introduces it.

Two guardrails keep routing honest. Pair every downgrade with an eval on real production samples, so “no measurable quality loss” is a measurement rather than a hope. And re-run the policy quarterly, because tier boundaries move every time a vendor ships.

The modifiers that quietly eat the savings

This is the section that separates a cost review from a cost audit. You can ship all three levers and still watch the bill climb, because these five modifiers sit outside the model price.

Tokenizer drift. Anthropic documents that Claude 4.7 and later models use a newer tokenizer that “produces approximately 30% more tokens for the same text”. Per-token price comparisons across model generations are therefore misleading. Baseline in cost per request, not cost per token.

Long-context tiers. Google prices Gemini 3.1 Pro Preview at $2.00 input below 200k tokens and $4.00 above it, with output going from $12.00 to $18.00. Anthropic goes the other way and includes the full 1M-token window at standard pricing on Claude 4.6 and later. A context-bloating retrieval bug costs you double at one vendor and nothing extra at another, which is a genuine architectural input.

Data residency and endpoint premiums. Anthropic applies a 1.1x multiplier on every token category when inference is pinned to US-only, and regional or multi-region endpoints on Bedrock and Google Cloud carry a 10% premium over global endpoints. Compliance is a legitimate reason to pay it. Paying it by accident, on a default nobody chose, is waste.

Server-side tool fees. These are billed outside the token meter entirely. Anthropic’s web search is $10 per 1,000 searches, code execution is $0.05 per container-hour beyond 1,550 free hours a month, and Managed Agents add $0.08 per session-hour on top of tokens. An agent that searches aggressively can spend more on tool calls than on inference.

Marketplace billing that destroys attribution. Buy through AWS Marketplace and usage is invoiced in Claude Consumption Units at $0.01 per CCU, with Anthropic’s docs stating that “AWS Cost Explorer shows aggregated CCU”. Your cloud cost tool sees one number. Per-model and per-team detail lives only in the vendor console. This is exactly the reconciliation gap our AI cost audit methodology is built to close.

Wire the guardrails so the savings hold

Optimization without enforcement decays. Three controls make it stick.

Set provider spend limits and know what they return. OpenAI supports organization-level and project-level hard limits; exceeded requests return a 429 with organization_spend_limit_exceeded or project_spend_limit_exceeded, and their docs warn that “enforcement is not instantaneous, so recorded spend can slightly exceed the configured amount”. Anthropic distinguishes two failure modes worth handling separately: the enforced tier cap returns a 429 with enforced_spend_limit_reached and no retry-after header, so SDK auto-retry will not recover, while a self-set limit returns a 400. If your alerting cannot tell those apart, your on-call will misdiagnose an outage as a rate limit.

Alert on token velocity, not monthly spend. A stripped cache breakpoint or a looping agent can burn a month’s budget in an afternoon, long before a month-end report notices. This is precisely the failure class our budget and alert validation work tests for, because most alert configurations have never been fired in anger.

Attribute at the gateway. Tag every call with team, feature, and request identifiers before it leaves your infrastructure, then reconcile those logs against the provider invoice. Without that, every number above is theory. This control is one of the domains scored in the FinOps maturity assessment framework, and the reason it scores badly so often is the reason we argue FinOps is a quality problem, not just a finance problem: teams buy the tool, skip the validation, and never learn the data was wrong.

Where to start on Monday

Pick the async workload with the highest monthly spend and move it to batch. That is worth 50% and needs no evaluation, no prompt rewriting, and no quality argument. Then instrument cache hit rate on your top three routes before touching a single cache_control flag, so you can prove the second lever paid for itself. Routing comes third, because it is the one that needs evals.

And re-check the prices. Every figure here carries a date for a reason.

If you would rather have an outside auditor reconcile your gateway logs against your token invoices, find which routes are overpaying, and hand back a governance plan that holds after the engagement ends, that is exactly what our AI & GPU Cost Governance QA engagement does. Get in touch for a free 30-minute review.