So we measured it. Same model, same prompt, the same call sent three ways, and the answer is clearer than we expected: the gateway's own software adds about 26 milliseconds. Everything else you might see is network distance — and that's in your hands, not ours.

The setup​

The trick to a fair latency test is holding everything constant except the thing you're measuring. So all three runs below hit the same upstream — OpenAI gpt-4o-mini — with the exact same prompt and parameters:

{
  "model": "gpt-4o-mini",
  "messages": [{ "role": "user", "content": "Reply with the single word: pong." }],
  "max_tokens": 5,
  "temperature": 0
}

Because every path ends at the same OpenAI model, the model's own response time is a shared constant across all three. Whatever difference is left over is the gateway, not the model.

The three paths:

1. OpenAI direct — call api.openai.com straight from the test machine. This is the baseline.
2. Local Acrux — the same call through an Acrux gateway running on localhost, which forwards to OpenAI.
3. Hosted Acrux — the same call through our hosted gateway at api.acruxcore.com, which forwards to OpenAI.

Each path ran 60 interleaved rounds (the paths take turns, so a passing network blip hits all three equally), after a warm-up that we discarded. Connections were pooled, so no path pays a fresh handshake on every call. We report the median and the p95/p99 tails, not the average — averages hide the spikes that actually annoy users.

The headline: the software adds ~26 ms​

Here is the clean comparison — OpenAI direct versus the same call through a gateway on the same machine. Same machine means the gateway's call to OpenAI travels the identical network path as the direct baseline, so the gap between them is purely the gateway's own code.

That's it: 26 milliseconds for routing, resolving your model name, applying your key, and logging the call. On a request that already takes ~800 ms of model time, the gateway is roughly 3% of what you were already going to wait.

"But the hosted number looks bigger" — here's why​

Run the same test against our hosted gateway and the number jumps to a median of 1221 ms. Before you read that as "the gateway is slow," look at where the extra time actually goes — because almost none of it is gateway code.

To prove it, we ran a second test that makes no model call at all: a plain round-trip to the hosted server (a 404), and a lightweight authenticated read. That isolates each layer without the model's response time on top. Here's the hosted 1221 ms broken into its real parts:

The two big slices have nothing to do with our software:

• The model call itself (~947 ms) is work you pay no matter what. Call OpenAI directly and you still wait for the model to think.
• The network hop (~150 ms) is the distance between the test machine and the server. We ran this from a developer laptop that sits far from the hosted box. A user who deploys their app near the gateway would see a fraction of this.

Only the last two slices are the platform's own doing: ~98 ms for authentication and a database lookup, and ~26 ms of gateway code. Put differently — the gateway software is about 2% of that 1221 ms.

Don't forget the tail​

Medians tell you the typical case, but users remember the slow one. Here's the full distribution:

Local Acrux tracks the OpenAI baseline closely all the way out to p95 — the gateway doesn't just add a small median cost, it adds a small tail cost too. The hosted p99 (3743 ms) is wider, and that widening is the network again: a long-distance hop has more room to occasionally stall than a loopback call does.

What this means for you​

A gateway earns its keep — provider routing, cost tracking, caching, budgets — and the fear that it taxes every request turns out to be mostly unfounded:

• The gateway's software cost is small — roughly 26 ms, a few percent of a typical model call.
• Most real-world latency is network distance, which you control by deploying the gateway near the app that calls it.
• The tail stays tight when the network is short.

If you're weighing whether to route your LLM traffic through a gateway, latency isn't the reason to hesitate. Deploy it close to your callers and the overhead disappears into the noise of the model call you were already waiting for.

The one-line difference​

LangSmith grew up around tracing and evaluation, with deep ties to the LangChain/LangGraph ecosystem, and later added prompt management.

Acrux Core is built around a runtime gateway plus prompt versioning as the spine, with tracing, a tool catalog, and evaluation layered on the same team-scoped model — one continuous line from authoring to measurement.

Prompt versioning​

Both let you store prompts and version them.

• Acrux Core treats every version as immutable and uses aliases (production, staging) that you move between versions. Your app fetches my-prompt@production at runtime via the SDK; promoting a new version changes behavior with no redeploy. Prompts can also bind a default model.
• LangSmith offers a prompt hub with commits and tags, and is especially convenient if you're already pulling prompts through LangChain.

If "change the live prompt without shipping code" is a core workflow, Acrux Core's alias model is built around exactly that.

The model gateway​

This is the biggest structural difference.

• Acrux Core ships an OpenAI-compatible gateway: one endpoint in front of OpenAI, Anthropic, Gemini, and any OpenAI-compatible provider (OpenRouter, Together, local servers). You bring your own keys (encrypted at rest), register public model names, and get routing, cost accounting, caching, virtual keys, and budgets. Your app can even send a stored-prompt reference and let the gateway render and route in one call.
• LangSmith does not position itself as a proxy/gateway; you typically call providers directly (often through LangChain) and send traces to LangSmith.

If you want a single audited, cost-visible path to every provider, that's a first-class feature in Acrux Core rather than something you assemble yourself.

Tracing​

• LangSmith has deep, mature tracing — rich support for nested runs, and automatic instrumentation when you use LangChain/LangGraph.
• Acrux Core traces every gateway call automatically (model, tokens, latency, cost) and accepts your own OTel-shaped spans via the SDK's trace() for whole chains. Traces can be grouped into sessions, and feedback attaches to traces and spans.

For LangChain-heavy apps, LangSmith's automatic instrumentation is hard to beat. For framework-agnostic apps, Acrux Core gives you tracing for free the moment you route through the gateway.

Tools​

• Acrux Core has a tool catalog: callable functions versioned exactly like prompts, attached to a prompt version, with a client-side runToolLoop in the SDK or a gateway-run HTTP executor.
• LangSmith leans on LangChain's large library of tool integrations.

Different philosophies: a governed, versioned in-house catalog versus a broad ecosystem of ready-made integrations.

Evaluation​

Both support datasets and experiments.

• Acrux Core builds datasets from real production feedback (thumbs up/down + comments on traces), then runs experiments that sweep version × model grids with an automatic production baseline.
• LangSmith has a rich, well-established evaluation suite with many built-in and custom evaluators.

LangSmith's eval tooling is broader today; Acrux Core's angle is tight coupling between the feedback you already collect and the datasets you evaluate against.

Hosting​

• Acrux Core is a standard Postgres + Node stack you can self-host; BYOK keeps provider keys under your control.
• LangSmith is primarily a managed cloud service (with enterprise self-hosting options).

Which should you pick?​

• Choose LangSmith if you're deep in the LangChain/LangGraph ecosystem and want the most mature tracing and evaluation available today.
• Choose Acrux Core if you want prompts, a provider gateway, tracing, tools, and evaluation as one integrated, self-hostable platform — and you value runtime prompt swaps and a single cost-visible path to every model.

Want to try the integrated story? The Quickstart gets you from sign-up to a traced, gateway-routed call in about ten minutes.