Exact-match gateway caching: latency and cost saved
A gateway virtual key can be given a cache window: set cacheTtlSeconds and
an exact-match repeat of a call — same model, same messages, same
temperature/max_tokens/top_p/stop — is served from Postgres instead
of the provider. We ran the same fixed set of prompts through it repeatedly
to see what a hit is actually worth, in milliseconds and in dollars.
The setup
Fifteen distinct one-word-answer prompts, sent through gpt-4o-mini with
temperature: 0 (the cache deliberately never stores a call with an
implicit or non-zero temperature — see the note below on why). Each of the
15 prompts was sent four times: the first send of a prompt is
necessarily a cache miss; the three repeats that follow are hits, since
nothing about the request changed. Sixty calls total, against a virtual key
created just for this run with a five-minute cache window, on a real running
gateway.
The latency gap
| n | median | p95 | |
|---|---|---|---|
| Cache miss (real provider call) | 15 | 902 ms | 2709 ms |
| Cache hit (Postgres lookup) | 45 | 9 ms | 10 ms |
A hit is about 100x faster than a miss here, and its p95 is barely wider than its median — that's the shape you'd expect from a single indexed lookup versus a network round trip to a model provider, where the tail is dominated by the provider's own variance, not the gateway's.
The cost gap
The cache doesn't just skip the network call — it skips the bill. A hit's
x-gateway-cost-usd header reads 0, and the gateway's own ledger records
cache_hit rows at zero cost rather than opening a spend-tracking
transaction at all:
| calls | total cost | |
|---|---|---|
| Misses (real spend) | 15 | $0.000063 |
| Hits (would-have-cost, at the measured miss rate) | 45 | $0.000189 |
| Actual spend, this run | 60 | $0.000063 |
At this run's 1-in-4 duplicate rate, caching cut spend by 75% — which is
just the duplicate share of the traffic (45 of 60 calls were repeats). The
absolute numbers above are tiny because gpt-4o-mini is cheap and the test
used 5-token replies; the ratio is what's portable. Route traffic through
the gateway where a meaningful share of calls repeat exactly — a support bot
answering the same handful of FAQs, a batch job re-processing overlapping
input — and that ratio is your cost reduction, whatever your real per-call
price is.
What actually gets cached
The cache key is a hash of model, messages (in order), temperature,
max_tokens, top_p, and stop — nothing more, nothing less. A few
specifics worth knowing before you rely on it:
temperaturemust be explicitly0. An omitted temperature is never cached, on purpose — the assumption is that if you didn't pin the temperature down, you don't want a stale answer standing in for a fresh sample.- Streaming calls are never cached, on either side — a streamed request is never looked up and never stored.
- The cache is per-team. Two teams sending the identical request get two separate entries; nothing about a hit is shared across tenants.
- It's still fully traced. A hit writes the same span a miss would, tagged so you can tell them apart, and still shows the real token counts — only the cost is zero and no provider was actually called.
What this means for you
- Turn it on for a virtual key that serves repeat traffic — the win scales with how often the exact same request recurs, not with volume on its own.
- Pin
temperature: 0deliberately if you want caching — it's the one field the gateway insists on before it will store anything. - Cost savings equal your duplicate rate, not a fixed percentage — this run's 75% came from sending each prompt four times; a lower repeat rate buys proportionally less.
Want to reproduce this? The exact numbers above come from one live run of
gateway-cache-bench.mjs against a running gateway — 15 fixed prompts, 4
passes each, a virtual key with cacheTtlSeconds set, reading the real
x-gateway-cache and x-gateway-cost-usd response headers rather than
inferring hit/miss from latency alone.