12 posts tagged with "LLM tracing"

Laminar (lmnr) is a Y Combinator S24 project that calls itself observability purpose-built for AI agents — a Rust ingest server, ClickHouse for spans, Quickwit for search, and an OpenTelemetry SDK that instruments fifteen-plus agent frameworks from one line of code — Vercel AI SDK, Claude Agent SDK, OpenAI Agents SDK, LangChain DeepAgents, Mastra, Pydantic AI, Browser Use, Stagehand, Playwright, LiteLLM, OpenCode and OpenHands among them. It is the most differently shaped tool we have compared AcruxCore against. Its project sidebar has dashboards, traces, evaluations, datasets, labeling, a SQL editor, playgrounds, a debugger and settings — and no prompt registry anywhere. That is not an oversight; it is the product being about agent runs rather than about the calls inside them. Both projects are Apache 2.0, so this is a comparison of shape, not of who is open.

Helicone is a proxy-first LLM observability tool — you point your API base URL at it and it sits in the request path, which makes it architecturally closer to AcruxCore's own gateway than most competitors in this series. So we built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on self-hosted Helicone, then tried to run the identical sequence we've run on every platform in this series: create the prompt, send a live call, inspect the trace, build a dataset, and call it from a script. On this self-hosted build, three of those steps hit real, reproducible errors on Helicone's side before we ever got a trace. We're publishing those errors, not routing around them — that's the most useful thing we found.

Opik is Comet's open-source LLM-ops platform, and unlike some competitors we've covered it ships with genuinely no login wall when self-hosted — you docker compose up and you're working inside a real workspace immediately. We built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on both platforms, then ran the identical sequence on each: create the prompt, version it, send a live call, inspect the trace, build a dataset, run an experiment, and call it from an SDK script.

Arize Phoenix is the OTel-native tracing-and-eval project a lot of teams reach for first, so it deserves a real answer, not a table copied from two docs sites. We built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on both platforms, then ran the identical sequence on each: create the prompt, version it, send a live call, inspect the trace, build a dataset, run an experiment, and call it from an SDK script.

Langfuse is the LLM-ops platform we get compared to most often, and it deserves a real answer, not a table copied from two docs sites. So we built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on both platforms, then ran the identical sequence on each: create the prompt, version it, send a live call, inspect the trace, build a dataset, run an experiment, and call it from an SDK script.

Most tool comparisons are written from docs and marketing pages. We didn't do that here. We ran nine LLM-ops platforms ourselves — LangSmith, Langfuse, PromptLayer, Arize Phoenix, Opik, MLflow, Helicone, and Laminar, plus our own AcruxCore as the baseline — and did the same thing on each one: create a prompt, version it, run it live with a real model key, inspect the resulting trace, and try to build an eval. Then we wrote a small script against each platform's own SDK and ran that too, and timed every one of them against a direct call to the provider.

Tracing and monitoring — what most people mean by "LLM observability tools" — is only one of nine angles below; prompt management, evals, guardrails, and tool-calling get the same hands-on treatment.

In the previous post we wrote the same tool-calling agent against a raw OpenAI-compatible client on both platforms and hand-rolled the loop. AcruxCore needed 105 lines to LangSmith's 63 — we lost that one, and said so.

This post is the other half: what happens when you use each platform's own abstraction instead of writing the loop yourself. LangChain's create_agent against the AcruxCore SDK's run_tool_loop. Same tool, same question, same model (openai/gpt-4o-mini via OpenRouter), both run live.

A tool call is where an agent stops being a text generator and starts touching the world. It is also where tracing earns its keep: when an agent gives a wrong answer, the question is almost never "what did the model say" — it is "what did the tool return, and how long did it take".

So we built the same tool twice and ran it for real. Same tool, same question, same model (openai/gpt-4o-mini through OpenRouter on both sides, so nothing hides behind a model difference). One run traced by LangSmith, one by AcruxCore. Then we did it again with streaming, which is where we found a bug in our own product.

Every number and every output below comes from a run we actually executed, and all three scripts are at the end of the post in full.

This is the AcruxCore leg of a hands-on comparison series. We ran the exact same steps — create a prompt, version it, generate a trace, try to build a dataset — against the hosted product ourselves, so the LangSmith, Langfuse, and PromptLayer write-ups have a fair baseline to compare against. No marketing framing here — just what the dashboard actually does. The main comparison post pulls the findings from all four platforms together.

Langfuse is one of the best-known open-source LLM engineering platforms — it covers prompt management, tracing (recording what an LLM app actually did, step by step), evaluation, and datasets. This is the Langfuse leg of a hands-on comparison series: we logged into Langfuse's hosted EU cloud with a real account and clicked through the whole loop ourselves — open a prompt, version it, inspect a real trace, and set up a dataset for testing. This post is a plain, factual account of what we found — the good and the rough edges. The main comparison post pulls the findings from Langfuse, LangSmith, PromptLayer, and AcruxCore together.