Semantic caching is the practice of returning a stored answer when a new prompt is semantically close to one you've already answered — instead of paying the model provider to generate the same thing again. Not an exact string match, but a meaning match. "What's your refund policy?" and "How do I get my money back?" are different strings and the same question. A semantic cache recognises that and replays the earlier answer at a cost of zero.
For a CTO, that's a latency win and an availability win. For a CFO, it's the single largest lever on an AI bill that most teams never pull. On repetitive production traffic, a well-tuned semantic cache typically eliminates 40–80% of provider calls — which flows straight to the invoice, because a cache hit costs nothing.
This piece explains what semantic caching is, why so much production traffic is cacheable, how the replay works (including streaming), the three things teams get wrong, and — honestly — when caching does not help.
Why repetitive traffic is 40–80% cacheable
The intuition that "every prompt is unique" is wrong for most production systems. Look at where the tokens actually go:
- Support and Q&A. A finite set of questions, asked a thousand different ways. Refunds, password resets, "is it down?" The long tail is real, but the head is enormous.
- RAG over a stable corpus. When the underlying documents don't change hour to hour, the same questions retrieve the same context and deserve the same answer. Re-embedding and re-generating on every hit is pure waste.
- Agents re-asking. Agent loops are the worst offenders. A planning agent re-derives the same sub-question across runs; a retry loop asks the identical thing three times in ten seconds. This is exactly the traffic that produces the runaway-bill horror stories.
The FinOps Foundation, in its guidance on managing GenAI spend, calls out caching as a first-order optimisation precisely because production traffic clusters far more tightly than teams expect. We won't put a fabricated number on your workload — but the 40–80% range is what repetitive traffic tends to yield, and you can measure your own hit rate before you commit to anything.
How the replay works
A semantic cache sits inline on the request path — for Tokenality, inside the governed proxy that every call already passes through. The flow on each request:
- Canonicalise the prompt. The system prompt and messages are joined into a single lookup text, in the same shape the provider would see.
- Embed it. That text is turned into a vector — a numeric fingerprint of its meaning.
- Similarity match. The cache looks for a prior entry whose embedding is close enough to this one, scoped to the same org and the same model.
- On a hit — replay. The stored response is returned directly. No provider call is made. Actual cost for the call is 0, and any budget reservation is released back to the envelope rather than debited.
- On a miss — proceed and store. The call goes upstream as normal, and its response is written to the cache for next time.
Streaming is replayed too. A cached answer doesn't have to arrive as one blob — it can be streamed back to the client so the user experience is identical to a live generation, just without the provider bill behind it.
The three things teams get wrong
Naïve caching is easy to build and easy to get subtly, expensively wrong. Three failure modes matter.
1. Staleness and correctness — the similarity threshold
The whole game is the threshold. Set it too loose and you serve "the capital of France is Paris" to someone who asked about the capital of Australia — a confident, wrong, cached answer. Set it too tight and your hit rate collapses to near zero and you've saved nothing.
The threshold is a correctness dial, not just a cost dial. And some traffic should never be cached regardless of similarity: anything time-sensitive ("what's my current balance?"), anything that must be freshly reasoned, anything where a stale answer is worse than a slow one. A serious cache lets you scope which traffic is eligible — you opt an org or a workload in, rather than caching everything by default and hoping.
2. PII in the cache key — the subtle leak
This is the one almost everyone misses. To match prompts, you have to key on their content. If you hash the raw prompt, then a prompt containing a customer's name, email, or account number puts that PII into your cache store — as the key, as the embedding text, sitting in your infrastructure indefinitely. You've built a governance control that quietly creates a data-residency problem.
Tokenality redacts PII out of the text before it's hashed or embedded. Our redactForCache step runs the same detectors as the fail-closed PII pre-flight, replacing each detected span with a type tag ([email], [ssn]) so raw PII never enters the cache store. It's fail-closed for privacy: if the detector faults, the content is dropped rather than cached in the clear. A useful side effect — two prompts that differ only in their PII collapse to the same key, which is exactly the behaviour you want from a cost-saving cache. "Refund order 4471 for Jane Doe" and "Refund order 4471 for John Smith" cache as the same governed request.
3. Attribution — a hit is a saving, not a disappearance
Here's the failure that shows up in the finance review. When a call is served from cache, the naïve implementation records nothing — no provider cost, so no row. Now your usage reporting undercounts real demand, and worse, the savings the cache produced are invisible. You can't defend a cost line you can't see.
A cache hit should land in the ledger as a first-class saving. In the Token Ledger, a hit is recorded with an actual cost of 0, its budget reservation released, and a cacheHitSavedUsd figure — the dollar amount the org would have paid had the call gone to the provider. That last field is the whole point: it turns "we cache stuff" into "the cache saved this team $X last month," a number a CFO can put on a board slide.
Cache hit vs cache miss, side by side
| | Cache miss | Cache hit |
|---|---|---|
| Provider call | Yes — full generation | None |
| Cost to you | Full provider cost | $0 |
| Latency | Model generation time | Lookup + replay (much lower) |
| Budget | Reservation settled against actual cost | Reservation released — not debited |
| Recorded in ledger | Tokens, cost, model, project, person | Cost 0 + cacheHitSavedUsd (the avoided spend) |
| Streaming | Streamed from provider | Streamed from cache — identical UX |
When caching does not help — honestly
Semantic caching is not a universal discount, and selling it as one is how teams end up disappointed or, worse, serving wrong answers.
- Highly unique prompts. If every request carries a large, distinct payload — a full contract, a unique code diff, a one-off document to summarise — there's nothing to match against. Hit rate near zero, no saving.
- Creative generation. When the point is a fresh, varied output — brainstorming, copywriting, "give me ten different taglines" — a cache that returns the same answer twice defeats the feature. Don't cache what's supposed to be non-deterministic.
- Rapidly changing ground truth. Live inventory, current prices, "what happened in the last hour." A cache here is a correctness bug wearing a cost-saving costume.
The honest framing: caching is a large lever on repetitive traffic and roughly a no-op on genuinely novel traffic. The skill is knowing which of your workloads is which — and scoping the cache to the first kind. Measure your hit rate on a real slice of traffic before you promise anyone a percentage.
See it on a real governed call
Semantic caching only earns trust when you can watch the accounting. In the Tokenality playground you can make a governed call, watch the same request come back from cache on the second try at zero cost, and see the hit land in the Token Ledger as a recorded saving — cost 0, budget untouched, avoided spend attributed. The cache and the ledger agree, which is the property that matters when finance asks where the savings went.