Full-cycle latency across six LLM-ops platforms, measured against real OpenAI

Every latency number we'd published before this measured one thing: the completion call. But a real request to any of these platforms is usually two round trips — fetch or resolve a stored prompt, then complete it — and most of them were benchmarked against OpenRouter, not the provider whose name is on the model.

So this one measures the full cycle, on real OpenAI billing, across all six platforms plus both of AcruxCore's calling modes — eight paths in one interleaved run. We also ran it four independent times, because a single 100-round run against a live third-party API turned out to not be enough to trust a single number, and that's worth showing rather than hiding.

The setup

All eight paths hit the same upstream — real api.openai.com, model gpt-4o-mini — with the same prompt and parameters (temperature: 0, max_tokens: 5). Each path does whatever that platform's stored-prompt feature actually requires (fetch a prompt, resolve a template, or nothing, if the platform has no such concept), then completes it, timed together as one number:

1. OpenAI direct — no platform in the loop. The baseline.
2. Opik tracked SDK** — Opik's track_openai() wrapper around a direct call.
3. MLflow AI Gateway** — routed through a local MLflow AI Gateway endpoint.
4. Langfuse OTel SDK — langfuse.get_prompt(), then a call instrumented with Langfuse's OpenTelemetry integration.
5. Helicone AI Gateway — routed through Helicone's AI Gateway.*
6. Phoenix OTel SDK — a call instrumented with Arize Phoenix's OpenTelemetry integration.
7. AcruxCore gateway — hub.prompts.render(), then hub.gateway.chat() routed through AcruxCore's gateway.
8. AcruxCore BYOK (gateway-free) — the same render call, then hub.gateway.chat() with a provider config that posts to OpenAI directly, skipping the gateway. See build a RAG agent without the gateway for the full BYO path.

Each run is 100 rounds plus 3 discarded warm-up rounds, with the eight paths interleaved in rotating order — the running order shifts every round, so no path is always first and a passing network blip hits all eight equally. Connections are pooled. We report medians with a 95% bootstrap confidence interval (5,000 resamples) on the gap against the OpenAI baseline: if that interval crosses zero, the gap isn't distinguishable from noise at this sample size, and we say so rather than quoting the point estimate as if it were exact.

Full script: full-cycle-latency-bench.py.

Median full-cycle latency, all eight paths

Median of 100 rounds, real OpenAI billing, most recent of four independent runs (raw data and all four runs' results are committed alongside the script).
OpenAI direct (reference) Other platform AcruxCore

All eight paths land within about 220ms of each other on the median — small next to the ~600–800ms the model call itself takes. The tail tells a fuller story:

Path	median	p95	p99
Opik tracked SDK**	810 ms	1982 ms	10999 ms
MLflow AI Gateway**	902 ms	1472 ms	3370 ms
Langfuse OTel SDK	954 ms	1487 ms	6002 ms
Helicone AI Gateway*	977 ms	1471 ms	2166 ms
Phoenix OTel SDK	973 ms	1366 ms	2695 ms
OpenAI direct (baseline)	978 ms	2706 ms	6402 ms
AcruxCore BYOK (gateway-free)	1022 ms	1458 ms	3688 ms
AcruxCore gateway	1032 ms	1676 ms	3546 ms

Notice the baseline's own p99 (6402ms) is worse than every platform's median and most of their p99s — a reminder that raw OpenAI, with nothing in front of it, has the widest tail here. That's normal live-network variance, not evidence any platform is "faster than the provider it calls."

Median, p95, and p99 are still only three numbers standing in for 100 real measurements each. Here's every one of those 800 rounds, plotted:

Every measured round (not just the median), most recent of four independent runs. Generated from `results.json` by plot_latency_charts.py.

The boxes overlap heavily — that's the point. Opik's box sits visibly lower than the rest, which is what the negative gap in the next section is measuring; every other platform's box straddles the baseline's dashed median line.

Is any of this gap real?

A median alone doesn't tell you whether a platform costs something or whether you're looking at noise. Bootstrapping the gap against the OpenAI baseline, per round (so a shared network blip cancels out), gives a confidence interval:

Platform	gap vs. OpenAI	95% CI	statistically real?
Opik tracked SDK**	−140 ms	[−287, −32]	Yes — doesn't cross zero
MLflow AI Gateway**	−81 ms	[−155, +34]	No
Langfuse OTel SDK	−22 ms	[−95, +84]	No
Helicone AI Gateway*	−6 ms	[−71, +111]	No
Phoenix OTel SDK	0 ms	[−122, +101]	No
AcruxCore BYOK	+39 ms	[−1, +155]	No
AcruxCore gateway	+63 ms	[−19, +172]	No

Only Opik's gap clears the zero line here, and it's negative — faster than the raw baseline. Everything else, AcruxCore included, is statistically indistinguishable from calling OpenAI with nothing in front of it, in this particular 100-round sample.

Opik's negative gap isn't a one-run fluke: it shows up in all four independent runs (−140ms to −168ms), and it's not a few outliers dragging the median either — plotting every round against the same-round baseline shows Opik coming back faster in 72–77% of individual rounds, every run:

Raw round-by-round latency, most recent run — same round index means the same round of the interleaved benchmark. The rolling median makes the separation visible through the network noise. Same script as the chart above.

That's a real, repeatable pattern. What it isn't, yet, is a fully explained one: an isolated side-by-side test of the two candidate causes — Opik's prompt rendering into a single merged message versus the baseline's separate system/user messages, and the track_openai() wrapper itself — reproduced only a small and inconsistent fraction of the gap, nowhere near its full size. The honest state of this one: real effect, confirmed cause not yet found. We're not going to publish a tidy explanation we haven't verified.

*Helicone's leg times completion only, not a real prompt fetch — its number is directly comparable to the baseline's shape, not the other platforms' two-round-trip cycles.

note

**Opik and MLflow store this prompt as flat text, not role-tagged chat messages — confirmed against each platform's own stored data. So both send one merged user message, not the system-plus-user split every other leg uses, which makes neither a fully apples-to-apples structural comparison. That's weaker evidence than it looks for "merged message causes the gap," though: MLflow shares the exact same shape and its gap crosses zero — if the message shape alone drove it, both should show it.

How stable is that gap, run to run?

Here's the part worth showing rather than skipping past: we ran the full eight-path benchmark four separate times, on different occasions, with identical code. Tracking AcruxCore's two modes against the OpenAI baseline across all four runs:

Gap vs. OpenAI baseline (ms), four independent 100-round runs of identical code. Vertical bars are 95% bootstrap confidence intervals; the dashed line at zero is "statistically identical to raw OpenAI." AcruxCore BYOK wasn't yet a leg in Run 1.
AcruxCore gateway AcruxCore BYOK

Run	AcruxCore gateway gap	Crosses zero?	AcruxCore BYOK gap	Crosses zero?
Run 1	+194 ms	No	— (not tested yet)	—
Run 2	+53 ms	No	−8 ms	Yes
Run 3	0 ms	Yes	−38 ms	Yes
Run 4	+63 ms	Yes	+39 ms	Yes

The gateway's point estimate moves from +194ms down to 0ms and back to +63ms across four runs of unchanged code — only two of the four even clear statistical significance. BYOK's gap crosses zero in every run it was measured.

That difference is itself informative. A 100-round bootstrap on top of a highly variable, 600–800ms live-network signal has real estimation width — commonly ±100–150ms per run at this sample size — which is enough to swallow a true underlying effect of a few tens of milliseconds. A small, real, and stable cost will still look like it's bouncing between 0 and 190ms across separate runs, purely from sampling noise, even though nothing about the code changed between them. BYOK skips the gateway's synchronous work — a rate-limit check, a budget-reserve transaction, and the request-persist step — while still recording a trace asynchronously, and that's the one number here that came back the same every time we asked.

The honest takeaway: if you need a single number for "how much does the gateway cost," don't take one run's point estimate — the range across four runs is closer to the truth than any one of them. We're publishing all four runs' raw data alongside the script for exactly that reason.

What this means for you

• All eight paths cluster within ~220ms of each other on the median — small next to the ~600–800ms the model call itself takes.
• Only one gap in this benchmark is consistently distinguishable from noise across runs: Opik's client-side wrapper measured faster than raw OpenAI. Part of that leg also isn't a strict apples-to-apples comparison — Opik sends one merged message instead of the split system/user shape every other leg uses (see the ** note above) — but isolated testing found the message shape explains only a small part of the gap, not the cause.
• AcruxCore's gateway adds a small cost that this method can't pin to an exact number — somewhere in the tens of milliseconds, from synchronous DB work on the request path, but not stable enough across independent runs to quote as a fixed figure.
• AcruxCore's BYOK mode is the one result that held up identically across every run: statistically indistinguishable from calling OpenAI directly, while still reporting a trace. If you want the gateway's tracing and versioned prompts without its synchronous request-path work, that's the trade BYOK makes — see route your calls through the gateway for the other side of that choice.
• Real third-party network latency is the dominant, most variable term in every path here — hundreds of milliseconds and swings of a similar size, next to a local cost measured in the tens.

Reproduce it

The script needs OPENAI_API_KEY, a personal AcruxCore API key, and local instances of whichever of Opik / MLflow / Langfuse / Phoenix / Helicone you want to include — it skips a platform cleanly if its env vars aren't set. Run it with python full-cycle-latency-bench.py; it prints the summary table and the bootstrap gap analysis to stdout, then writes results.json. Run it more than once before trusting any single number, for the reason above.

Full script and all four runs' raw data: scripts/comparison/shared/ on GitHub. Both charts above are generated straight from results.json by plot_latency_charts.py in the same folder — point it at any run's results.json to regenerate them.

For the narrower question of what AcruxCore's own gateway costs against a BYO call on the same machine, see how much overhead does an LLM gateway add?