Opik alternative: guardrails, PII and online evaluation
Opik is Comet's open-source LLM-ops platform, and unlike some competitors we've covered
it ships with genuinely no login wall when self-hosted — you docker compose up and
you're working inside a real workspace immediately. We built the same prompt —
vip-support-triage, a support agent that changes tone for VIP customers and lists
their open tickets — on both platforms, then ran the identical sequence on each: create
the prompt, version it, send a live call, inspect the trace, build a dataset, run an
experiment, and call it from an SDK script.
Every paired screenshot below comes from the exact same prompt and the exact same
customer message, sent through the exact same downstream model
(openai/gpt-4o-mini via OpenRouter) on both platforms. Opik's self-hosted instance
had no login screen and no credentials to configure, so this comparison was run as an
anonymous local user — exactly what a reader self-hosting it for the first time would
see. Where a step is genuinely one-sided — no equivalent screen exists on the other
product — we say so instead of padding it out. License, pricing, team structure, and
community stats live on the compare page instead of here — they were
always tables, and a price change there is one edit instead of three.
At a glance
| Aspect | Opik | AcruxCore | Winner |
|---|---|---|---|
| Dataset creation | Add-to-dataset from any trace, inline dataset creation | From real span-level trace feedback | Depends |
| Prompt templating | Flat {{variable}} only, but real Diff view + environment labels | Real nunjucks {% if %} / {% for %} logic, real Diff tab | AcruxCore |
| Playground | Latency + tokens shown inline, no cost, no cache | Cost, cache, latency shown inline | AcruxCore |
| Tracing depth | Span tree via client-side SDK instrumentation | Single automatic span | Depends |
| Request-path gateway | None — ingests a trace after your own call | Built in — routing, caching, budgets | AcruxCore |
| Tool catalog | No catalog at all — "Agent playground" needs a live connected process | Versioned catalog, real executed calls, analytics | AcruxCore |
| SDK trace capture | Wrap a client with track_openai() | Automatic side effect of the gateway call | AcruxCore |
| Measured overhead | +2 to +22 ms across three 100-round runs | +9 to +31 ms across the same three runs | Opik, barely |
| Time-to-first-trace | Only via the SDK path, after wrapping a client | Zero code, first call | AcruxCore |
License, pricing, team structure, security, and community stats: see AcruxCore vs Opik on the compare page.
Full breakdown, screenshots, and the verdict below.
Evaluation & datasets
Opik's "Add to → Dataset" works straight from a trace's detail panel — broader than a span-only restriction we hit on another platform — and creating a brand-new dataset is a one-step inline dialog from that same menu.
Show screenshot: Add to dataset dialog on Opik
AcruxCore's dataset — built from a real feedback row, since the vip-support-triage trace's own single-span feedback wasn't eligible — is one sentence here; see the walkthrough for the screenshot.
Running an experiment against that dataset from the UI goes through "Open in
Playground" — the dataset's rows load as bindable variables
({{input.messages.1.content}}-style paths), then Run. Clicking "Run an experiment"
directly instead explicitly says "Use this dataset to run an experiment using the
Python SDK" — confirming the SDK is the primary path for a scripted experiment, with
the Playground as the interactive alternative.
Show screenshot: dataset-bound experiment run in Opik's Playground
AcruxCore's Run report for that dataset against gpt-4o-mini — a plain, unscored manual run — is one sentence here; see the walkthrough for the screenshot.
| Feature | Opik | AcruxCore |
|---|---|---|
| Dataset creation | From any trace directly, inline "create new dataset" in the same dialog | From trace feedback only — and only span-level feedback |
| Experiments | Interactive via Playground, or scripted via the Python SDK (UI explicitly defers to it) | Version × model sweep with an automatic baseline, run from the UI |
| Feedback scores | Thumbs-up/down "Human review," visible in the dataset row copied over | Span-level feedback, gates dataset eligibility |
Prompt authoring & versioning
Opik's Prompt library offers a "text prompt" or a "chat prompt" — both are flat
{{variable}} substitution, confirmed hands-on: there is no {% if %} or {% for %}
anywhere. We flattened the VIP-branch note and the two-item ticket list into plain
text, the same way an earlier comparison had to for Langfuse, keeping only
{{company}} and {{customer_message}} as real variables.
Show screenshots: prompt editor and versions on both platforms
AcruxCore's own prompt editor — real nunjucks conditionals and loops rendering VIP status and the ticket list inline — is one sentence here; see the walkthrough for the picture.
Unlike a competitor we compared earlier that had no diff feature at all, Opik does have a real version comparison — we edited the prompt to create v2 and used its "Diff" button:
Show screenshot: version diff panel on Opik
It also has an environment-labelling control ("Deploy to") that tags a specific
version as production, staging, or development — functionally close to Acrux
Core's aliases, just without the runtime "fetch by alias" SDK call:
Show screenshot: Deploy to / environment labels on Opik
| Feature | Opik | AcruxCore |
|---|---|---|
| Conditional templating | Flat {{variable}} only — verified hands-on, no {% if %}/{% for %} | Real nunjucks {% if %}/{% for %}, rendered server-side |
| Version comparison | Real "Diff" view, whole-message red/green (not word-level) | Standing Diff tab, unified word-level diff between any two versions |
| Live/staging labels | "Deploy to" tags a version production/staging/development | production / staging aliases your app fetches at runtime |
Sending a live call
Opik's Prompt playground accepts OpenRouter directly as a named provider — paste a key, search any OpenRouter-routed model by name. Both playgrounds produced the identical VIP reply from the identical inputs.
Show screenshots: Playground run on both platforms
| Feature | Opik | AcruxCore |
|---|---|---|
| Playground | Runs against any OpenRouter-routed model, shows latency + tokens inline, no cost | Runs against a stored prompt reference, shows cost/cache/latency inline |
| Requires a real key | Yes | Yes |
Tracing & observability
Confirmed hands-on: running the prompt from Opik's Playground left the project's Logs
tab at "No traces yet" — the Playground alone does not trace, the same finding as an
earlier comparison. A real trace only appeared after calling the SDK-wrapped client,
producing a two-level span tree: an outer chat_completion_create trace wrapping an
inner LLM span with model, provider, tokens, and cost.
Show screenshots: trace detail on both platforms
| Feature | Opik | AcruxCore |
|---|---|---|
| Trace shape | Span tree (outer trace + inner LLM span) | Single span per gateway call |
| How it's produced | Client-side track_openai() wrapper | Automatic — a side effect of the gateway call |
| Playground produces a trace | No — confirmed, Logs stayed empty after a Playground run | Yes — every Playground run is traced |
Where the platform sits — in the request path, or beside it
AcruxCore's gateway sits in the request path: every call is routed through it, so BYOK provider selection, caching, budgets, and virtual keys all apply before the provider is ever called, and the trace is a side effect of that same hop. Opik sits beside the request path too — like the other observability-first platforms we've compared, it ingests a trace after your own client made the call, so there's nothing to route, cache, or budget against on its side.
| Feature | Opik | AcruxCore |
|---|---|---|
| Where it sits | Beside the request path — call providers yourself, trace is reported after | In the request path — every call routes through it |
| BYOK, caching, budgets, virtual keys | Not applicable — no request path to apply them to | Built in, applied before the provider call |
How tools are handled — schema registry, or execution
Opik has no tool catalog concept at all. Its closest surface, "Agent playground," is
not a schema-definition UI — it's a live-connection debugger: you add
@opik.track(entrypoint=True) to a running agent's own code and run a connector
command in your terminal, and Opik waits for that process to connect.
Show screenshots: Agent playground on Opik, tool analytics on AcruxCore
With nothing connected, the page just sits at "Disconnected" showing setup copy — this is weaker than even the placeholder "Create LLM Tool" schema dialog a prior comparison found on another platform, since that at least produced a stored, reusable JSON Schema object from the UI with zero code. AcruxCore's Tools page — a dozen real, versioned tools — is one sentence here; see the walkthrough for the screenshot.
| Feature | Opik | AcruxCore |
|---|---|---|
| Tool catalog | None — no schema object, no list view, no per-tool page | Persistent, versioned catalog with its own page |
| Agent visibility | Requires a live process connected via code + terminal command | Real, gateway-executed calls, traced automatically |
| Tool analytics | Not available | Calls, error rate, and latency per tool |
SDK & developer experience
Opik wraps a client you already own; the trace only appears once that wrapped client
makes a call. AcruxCore's SDK has dedicated surfaces — hub.prompts.render() then
hub.gateway.chat() — and the trace is automatic.
client = track_openai(
OpenAI(api_key=OPENROUTER_KEY, base_url="https://openrouter.ai/api/v1"),
project_name="Default Project",
)
response = client.chat.completions.create(
model="openai/gpt-4o-mini", temperature=0, max_tokens=256,
messages=[{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_MESSAGE}],
)
rendered = await hub.prompts.render("vip-support-triage", "production", VARIABLES)
result = await hub.gateway.chat(
rendered.model, rendered.messages,
temperature=0, max_tokens=256,
prompt_version_id=rendered.version_id,
)
Both produced a real trace within seconds of the call returning:
Show screenshot: SDK-produced trace on Opik
Full scripts: op_trace_run.py and acx_sdk_run.py.
AcruxCore's SDK-produced trace is one sentence here; see the walkthrough for the screenshot.
| Feature | Opik | AcruxCore |
|---|---|---|
| SDK model | Wrap your own client (track_openai(OpenAI(...))) | Dedicated hub.prompts / hub.gateway surface |
| Getting a trace | Automatic once the wrapped client is called | Automatic side effect of hub.gateway.chat() |
Latency overhead — measured
We timed the identical call three ways, interleaved in rotating order over 100 rounds
so a network blip hits all three equally: a raw direct call to the provider (baseline),
the same call wrapped in Opik's track_openai() client, and the same call through
AcruxCore's gateway. Every leg ends at gpt-4o-mini on api.openai.com with the same
key and body, and the script refuses to start unless the gateway model resolves to a
native OpenAI credential — so all three provably share one upstream. This measures two
different kinds of overhead — Opik's is client-side instrumentation cost, AcruxCore's is
an extra network hop that buys routing and caching — not a rigged head-to-head.
One run is not a result: the median gap moves enough between runs that a single number would mislead. So it ran three times.
| Run | Direct-call median | Opik gap | AcruxCore gap |
|---|---|---|---|
| 1 | 617 ms | +22 ms, CI [+7, +47] | +22 ms, CI [+10, +42] |
| 2 | 625 ms | +2 ms, CI [−13, +18] | +9 ms, CI [−6, +27] |
| 3 | 612 ms | +4 ms, CI [−15, +26] | +31 ms, CI [+5, +45] |
Opik's overhead lands between +2 ms and +22 ms, AcruxCore's between +9 ms and +31 ms. Opik's was never the higher of the two, though the two were identical in the first run. In two of the three runs both confidence intervals cross zero, so neither path is reliably distinguishable from calling the provider directly at this sample size.
Full script: latency_bench.py.
For a broader run — six platforms in one interleaved benchmark, the stored-prompt fetch timed alongside the completion, and four independent runs to check how stable the numbers are — see full-cycle latency across six LLM-ops platforms.
Friction hit during this run
Real friction hit while doing this comparison, not a symmetric wish list.
Opik
- 👍 No login wall at all on self-host — straight into a working project.
- 👍 OpenRouter is a first-class named provider in the Playground, no custom-base-URL workaround needed.
- 👍 "Add to dataset" works directly from any trace, with inline "create new dataset" in the same dialog.
- 👎 The Prompt playground doesn't produce a trace — only a wrapped SDK call does.
- 👎 Running an experiment from a dataset explicitly defers to the SDK; the UI-only path is the Playground, not a real batch experiment runner.
- 👎 The Agent playground requires editing your own agent's code and running a terminal connector command just to see it — no schema-only path at all.
AcruxCore
- 👍 Nothing to instrument —
hub.gateway.chat()(or the Playground) writes the trace as a side effect, no wrapper client. - 👍 Nunjucks conditionals and loops are real template logic, not something you flatten by hand first.
- 👎 A model with no registered per-1M rate shows a blank
—cost instead of$0or an estimate, until you register one.
Time-to-first-trace: AcruxCore gets there on the first call with zero tracing-specific code. Opik gets there too, but only through the SDK path, after wrapping a client.
What Opik does that AcruxCore doesn't
Sourced from Opik's own dashboard and docs, not from this post's earlier aspects — those carry the same bias any of our own feature list would.
Guardrails — a Topic guardrail and a PII guardrail, configurable per project.
Opik's "Set a guardrail" panel lets you enable a topic-restriction check (with a
sensitivity slider and a comma-separated restricted-topics list) and a PII check that
flags specific categories — credit card number, phone number, email, and more — each
with its own sensitivity threshold, plus a ready-to-run Python snippet using
opik.guardrails. AcruxCore has no equivalent: nothing inspects a call's input or
output for restricted topics or personal data before or after it reaches the model.
Show screenshot: guardrails panel on Opik
Test suites — a dedicated pre-deployment regression object. Distinct from Experiments: import test cases from a CSV or JSON file, or define them in the SDK, each with an expected output and a scoring method, framed by Opik itself as regression testing rather than dataset-driven experimentation.
Show screenshot: Test suites on Opik
AcruxCore has no equivalent to guardrails or a separate pre-deployment test-suite object. It does have its own rule-based online evaluation — a judge (built-in or your own prompt) scoring every matching live trace as it lands, not just a dataset run on demand.
Is AcruxCore an Opik alternative?
For the day-to-day loop of running a prompt in production, yes. For pre-deployment testing, not yet.
Opik has no tool catalog at all, its experiments default to the SDK rather than the UI, and its Playground produces no trace — only a wrapped SDK call does, which makes for a genuinely confusing first hour if you are evaluating it by clicking around.
Two things you would lose are real, and we will not talk around them: Opik's Topic and PII guardrails, configurable per project, and its Test suites — a dedicated pre-deployment regression object that nothing else in this series has an equivalent of. If either is the reason you run Opik, we are not a replacement for it.
Verdict
| Opik | AcruxCore | |
|---|---|---|
| Strongest at | Guardrails and PII detection, dedicated test suites, self-host with zero setup friction, real diff/deploy on prompts | A request-path gateway with routing, caching, and budgets on every call; tools that actually execute and get measured; datasets built from real feedback; rule-based online evaluation with a real judge model and custom judge prompts |
| Weakest at | No tool catalog at all; experiments default to the SDK, not the UI; Playground doesn't produce a trace | No guardrails, no dedicated test-suite object |
| Pick it if | You want guardrails and regression test suites layered onto a client you already own | You want the provider call itself — gateway, tools, feedback-driven datasets, and online scoring rules — traced for free, with nothing bolted on after the fact |
Opik's guardrails and test suites (What Opik does that AcruxCore doesn't above) are real, and neither is something AcruxCore has an answer for today. But on the loop that runs a production LLM app day to day, this comparison landed on AcruxCore's side: Opik has no tool catalog at all (How tools are handled above), and its Playground doesn't produce a trace, only a wrapped SDK call does (Tracing & observability above). AcruxCore's gateway, tool catalog, and feedback-driven datasets stay in one integrated flow. See the full picture, including license, pricing, and team structure, on the compare page.
Want to run this yourself? The Quickstart gets you from sign-up to a traced, gateway-routed call in about ten minutes.