Optimization
Model routing
Automatically sending each request to the cheapest model that can handle it, instead of defaulting everything to one expensive model. The routing decision happens before the call, based on the task.
Example
Simple "summarize this" requests route to a $0.60-per-million mini model; complex "analyze this contract for risk" requests route to a $15-per-million flagship. If 80% of traffic is simple, routing can cut the model bill by more than half without users noticing.
Related terms
Model tier
The rough class of a model by capability and price. Providers offer a range — small/fast/cheap up to large/slow/expensive — and the price gap between tiers is often 10–30×.
Provider arbitrage
Choosing among providers (or hosts) that serve comparable models at different prices, and sending traffic to the cheaper one. The same open model is often available from several vendors at meaningfully different rates.
Semantic caching
Serving a stored answer when a new request means the same thing as a previous one — even if the wording is different — so the model never runs and the provider is never billed. Unlike prompt caching (a provider discount), a semantic cache hit costs nothing at the provider.