How much overhead does an LLM gateway add?

Putting a gateway in front of your model providers buys you a lot: one endpoint for every provider, cost accounting, caching, virtual keys, and budgets. But it raises an obvious worry — am I paying for that with latency? Every request now takes an extra hop, and for a user-facing app, milliseconds matter.

So we measured it. Same model, same prompt, the same call sent three ways, and the answer is clearer than we expected: the gateway's own software adds about 26 milliseconds. Everything else you might see is network distance — and that's in your hands, not ours.

The setup

The trick to a fair latency test is holding everything constant except the thing you're measuring. So all three runs below hit the same upstream — OpenAI gpt-4o-mini — with the exact same prompt and parameters:

{
  "model": "gpt-4o-mini",
  "messages": [{ "role": "user", "content": "Reply with the single word: pong." }],
  "max_tokens": 5,
  "temperature": 0
}

Because every path ends at the same OpenAI model, the model's own response time is a shared constant across all three. Whatever difference is left over is the gateway, not the model.

The three paths:

1. OpenAI direct — call api.openai.com straight from the test machine. This is the baseline.
2. Local Acrux — the same call through an Acrux gateway running on localhost, which forwards to OpenAI.
3. Hosted Acrux — the same call through our hosted gateway at api.acruxcore.com, which forwards to OpenAI.

Each path ran 60 interleaved rounds (the paths take turns, so a passing network blip hits all three equally), after a warm-up that we discarded. Connections were pooled, so no path pays a fresh handshake on every call. We report the median and the p95/p99 tails, not the average — averages hide the spikes that actually annoy users.

note

The local gateway ran in development mode (an unoptimized dev server). That makes its overhead a slight over-estimate, so the real software cost is, if anything, a little lower than the number below.

The headline: the software adds ~26 ms

Here is the clean comparison — OpenAI direct versus the same call through a gateway on the same machine. Same machine means the gateway's call to OpenAI travels the identical network path as the direct baseline, so the gap between them is purely the gateway's own code.

Median of 60 interleaved rounds. The gap is +26 ms — the gateway's software cost.

That's it: 26 milliseconds for routing, resolving your model name, applying your key, and logging the call. On a request that already takes ~800 ms of model time, the gateway is roughly 3% of what you were already going to wait.

"But the hosted number looks bigger" — here's why

Run the same test against our hosted gateway and the number jumps to a median of 1221 ms. Before you read that as "the gateway is slow," look at where the extra time actually goes — because almost none of it is gateway code.

To prove it, we ran a second test that makes no model call at all: a plain round-trip to the hosted server (a 404), and a lightweight authenticated read. That isolates each layer without the model's response time on top. Here's the hosted 1221 ms broken into its real parts:

The hosted 1221 ms, decomposed. The gateway's own code is the thin bar on the right.

The two big slices have nothing to do with our software:

• The model call itself (~947 ms) is work you pay no matter what. Call OpenAI directly and you still wait for the model to think.
• The network hop (~150 ms) is the distance between the test machine and the server. We ran this from a developer laptop that sits far from the hosted box. A user who deploys their app near the gateway would see a fraction of this.

Only the last two slices are the platform's own doing: ~98 ms for authentication and a database lookup, and ~26 ms of gateway code. Put differently — the gateway software is about 2% of that 1221 ms.

tip

The portable takeaway isn't the 1221 ms — that number is specific to one laptop's distance from one server. It's the 26 ms. Wherever you run, the gateway's code adds tens of milliseconds; the rest is network distance you control by deploying close to your callers.

Don't forget the tail

Medians tell you the typical case, but users remember the slow one. Here's the full distribution:

Path	median	p95	p99
OpenAI direct	801 ms	1050 ms	1318 ms
Local Acrux	827 ms	1076 ms	1540 ms
Hosted Acrux	1221 ms	2013 ms	3743 ms

Local Acrux tracks the OpenAI baseline closely all the way out to p95 — the gateway doesn't just add a small median cost, it adds a small tail cost too. The hosted p99 (3743 ms) is wider, and that widening is the network again: a long-distance hop has more room to occasionally stall than a loopback call does.

What this means for you

A gateway earns its keep — provider routing, cost tracking, caching, budgets — and the fear that it taxes every request turns out to be mostly unfounded:

• The gateway's software cost is small — roughly 26 ms, a few percent of a typical model call.
• Most real-world latency is network distance, which you control by deploying the gateway near the app that calls it.
• The tail stays tight when the network is short.

If you're weighing whether to route your LLM traffic through a gateway, latency isn't the reason to hesitate. Deploy it close to your callers and the overhead disappears into the noise of the model call you were already waiting for.

note

Want to reproduce this? Every path used the same model, prompt, and parameters shown above, over pooled connections, 60 interleaved rounds. Point the same script at your own gateway and provider to measure your own numbers.