Optimization
Semantic caching
Serving a stored answer when a new request means the same thing as a previous one — even if the wording is different — so the model never runs and the provider is never billed. Unlike prompt caching (a provider discount), a semantic cache hit costs nothing at the provider.
Example
One user asks "What's your refund policy?" and another asks "How do I get my money back?" A semantic cache recognizes these as the same question and returns the stored answer to the second user at zero provider cost. On repetitive production traffic, a well-tuned semantic cache commonly removes a large share of provider calls.
Related terms
Prompt caching
A provider feature that stores the repeated front portion of your prompt so you don't pay full price to send it again. You mark the stable part (instructions, examples, a document); the provider bills it at a discount on subsequent calls. See cached tokens.
Cached tokens
Input tokens the provider recognizes from a previous request and serves at a steep discount instead of full price. Distinct from a full cache hit — here the model still runs, it just doesn't re-charge full rate for the repeated part of your prompt.
Model routing
Automatically sending each request to the cheapest model that can handle it, instead of defaulting everything to one expensive model. The routing decision happens before the call, based on the task.