Experiments & Runs API (Phase 5 E3)

All endpoints verified working via curl against a real dev server (real Postgres, a real BullMQ worker process — apps/worker — consuming a real local Redis, and a real OpenAI account behind the gateway connection — no mocked fetch, no mocked queue). Document updated only after curl confirmation, per CLAUDE.md's API-reference rule — except the prompt-mismatch-warning field's shape, documented from source; see "Prompt-mismatch warning" below.

An experiment ties a dataset (examples to evaluate against) to a (prompt-version × model) grid to sweep. Starting a run freezes the dataset's examples into exampleSnapshot, resolves the full grid — including an automatic baseline cell per model (the prompt's production alias if one exists, otherwise its latest committed version), added whenever the experiment names a promptId and the resolved baseline version isn't already an explicit version_ids entry — and enqueues a BullMQ Flow (one finalize job with one cell job per grid × example pair). Each cell job renders the prompt version against the example's input variables and calls the gateway in-process (GatewayService.complete), so budgets, rate limits, and the cost ledger apply exactly like live traffic. The finalize job (a BullMQ Flow parent) only runs once every child cell has settled, and marks the run succeeded or failed.

Auth: requireAnyAuth (session cookie or personal/team API key) on every route. GET routes are open to any member. POST /experiments, DELETE /experiments/:id, POST /experiments/:id/runs, DELETE /runs/:id and POST /runs/:id/rejudge additionally require requireRole('owner', 'admin', 'editor') — starting a run enqueues one paid provider call per (version × model × example) cell, and a re-judge one paid judge call per result, so neither is a read-only action. A team-scoped API key holds no role and is refused on those five; use a personal key from an editor account. Team-scoped throughout.

Verified with apps/api running via npm run dev and a real, separately booted apps/worker process (node dist/index.js), both pointed at the same Redis instance so the API's enqueued jobs are actually picked up by the worker — not an in-process/inline stub.

POST /api/v1/experiments

Creates an experiment: a dataset to evaluate, an optional prompt under test (enables the automatic baseline cell), and the explicit (version × model) grid to sweep. version_ids and models must each have at least one entry.

judge_model is required: the model that grades every result in every run of this experiment. It must be a model your team has registered under Gateway → Models (the same publicName you would send to the gateway), and it does not have to be one of the models in models. The server has no default judge, so a request without it is a 400. It is stored on config.judgeModel, so every run of the experiment is graded by the same judge.

Two more fields exist alongside those, both optional:

alias (request) — which alias's version the automatic baseline cell (added when prompt_id is set and that version isn't already among the explicit version_ids) points to. Omitted, the baseline resolves to the prompt's production alias if one exists, otherwise its latest committed version. Has no effect when prompt_id is omitted, since no baseline cell is ever injected. See "Alias-based baseline" below.
promptMismatchWarning (response only) — a non-blocking notice returned when the dataset's examples were sourced from a prompt other than prompt_id. See "Prompt-mismatch warning" below.

curl
TypeScript
Python

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1", "prompt_id": "07494047-7380-4324-acf9-069a40265323", "name": "greeting sweep", "version_ids": ["35ed29ec-412d-45bb-8420-66dfe02eb0d2"], "models": ["gpt-4o-mini"], "judge_model": "gpt-4o-mini"}'

const experiment = await hub.experiments.create({
  datasetId: '20e0308d-ec9c-40ac-8cd9-c20b4bb95df1',
  promptId: '07494047-7380-4324-acf9-069a40265323',
  name: 'greeting sweep',
  versionIds: ['35ed29ec-412d-45bb-8420-66dfe02eb0d2'],
  models: ['gpt-4o-mini'],
  judgeModel: 'gpt-4o-mini',
});

experiment = await hub.experiments.create(
    dataset_id='20e0308d-ec9c-40ac-8cd9-c20b4bb95df1',
    version_ids=['35ed29ec-412d-45bb-8420-66dfe02eb0d2'],
    models=['gpt-4o-mini'],
    judge_model='gpt-4o-mini',
    prompt_id='07494047-7380-4324-acf9-069a40265323',
    name='greeting sweep',
)

Response (status 201) — runs is empty right after creation:

{
  "id": "320a1e06-26cd-4e92-ab2d-d5eaf05db1b1",
  "teamId": "6b319ddb-bae8-469e-9af1-ffac0d4df846",
  "datasetId": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1",
  "promptId": "07494047-7380-4324-acf9-069a40265323",
  "name": "greeting sweep",
  "config": { "models": ["gpt-4o-mini"], "judgeModel": "gpt-4o-mini", "versionIds": ["35ed29ec-412d-45bb-8420-66dfe02eb0d2"] },
  "createdBy": "9bf448bd-5c2f-43d0-be81-08b117195b8d",
  "createdAt": "2026-10-13T13:06:22.237Z",
  "runs": []
}

Error responses

Missing judge_model (status 400) — there is no server-side default:

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1", "prompt_id": "07494047-7380-4324-acf9-069a40265323", "version_ids": ["35ed29ec-412d-45bb-8420-66dfe02eb0d2"], "models": ["gpt-4o-mini"]}'

{ "error": { "code": "VALIDATION_ERROR", "message": "judge_model is required: name a model registered under Gateway → Models." } }

A judge_model your team has not registered (status 400) — checked before anything is written, and a model registered by another team does not count:

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1", "prompt_id": "07494047-7380-4324-acf9-069a40265323", "version_ids": ["35ed29ec-412d-45bb-8420-66dfe02eb0d2"], "models": ["gpt-4o-mini"], "judge_model": "claude-judge"}'

{ "error": { "code": "VALIDATION_ERROR", "message": "Model 'claude-judge' is not registered. Add it under Gateway → Models." } }

Missing version_ids (status 400):

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1", "models": ["gpt-4o-mini"], "judge_model": "gpt-4o-mini"}'

{ "error": { "code": "VALIDATION_ERROR", "message": "Required" } }

A prompt_id that is not one of your team's prompts (status 404) — the same answer whether it belongs to another team or to nobody, so the endpoint never confirms that an id names a real prompt somewhere:

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "<your dataset>", "prompt_id": "<another team's prompt>", "version_ids": ["<a version>"], "models": ["gpt-4o-mini"], "judge_model": "gpt-4o-mini"}'

{ "error": { "code": "NOT_FOUND", "message": "Prompt not found." } }

A version_ids entry outside your team is the same, and every entry is checked — a version of your own prompt in the same array does not carry a foreign one through:

{ "error": { "code": "NOT_FOUND", "message": "Prompt version not found." } }

Custom judge prompt (`judge_prompt_id` field)

judge_prompt_id is optional: one of your team's prompts, used instead of the built-in judge prompt to grade every result. It must have at least one committed version. It is stored on config.judgePromptId. Each run resolves it when it starts, to the prompt's production alias or else its latest version, and grades the whole run with that version.

The prompt's messages are templates with four variables: {{ output }}, {{ criteria }}, {{ overallFeedback }}, and {{ history }} (earlier turns as JSON). A missing value is an empty string. The platform's JSON output instruction is always appended after them. When an example has an image, the image parts are added after the text of the template's last user message.

A custom judge grades every example, including one with no criteria and no overall feedback; the built-in judge leaves such an example unscored with the reason not judged. If the template fails to render, the result is unscored with the reason judge prompt error: … and no judge call is made. See Write a custom judge prompt.

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "5c5cdfac-690c-4514-8c60-5a4b196917f5", "prompt_id": "de3f1cb7-9536-4a15-9f84-03eed3a85b34", "name": "receipts with custom judge", "version_ids": ["c39d2551-0a41-48c7-9bcd-1ff2d08153c5"], "models": ["gpt-4o-mini-or"], "judge_model": "gpt-4o-mini-or", "judge_prompt_id": "409e4e4e-755e-466f-b01a-4522a6deca28"}'

Response (status 201):

{
  "id": "aa67a319-3424-4b16-9562-c3001b45ae13",
  "teamId": "8d3ceb9b-39d0-463b-8b91-aa2ef20ac9ba",
  "datasetId": "5c5cdfac-690c-4514-8c60-5a4b196917f5",
  "promptId": "de3f1cb7-9536-4a15-9f84-03eed3a85b34",
  "name": "receipts with custom judge",
  "config": { "models": ["gpt-4o-mini-or"], "judgeModel": "gpt-4o-mini-or", "versionIds": ["c39d2551-0a41-48c7-9bcd-1ff2d08153c5"], "judgePromptId": "409e4e4e-755e-466f-b01a-4522a6deca28" },
  "createdBy": "18c76b52-ee0d-4001-be8e-c29976488fbb",
  "createdAt": "2026-10-13T18:30:05.078Z",
  "runs": []
}

A judge_prompt_id that is not one of your team's live prompts (status 400) — the same answer for another team's prompt, a deleted one, or an id that names nothing:

{ "error": { "code": "VALIDATION_ERROR", "message": "judge_prompt_id does not refer to a prompt in this team." } }

A judge prompt with no committed version (status 400), since every judge call would fail:

{ "error": { "code": "VALIDATION_ERROR", "message": "Judge prompt 'empty judge 1791916204' has no committed version yet. Commit its judge template first." } }

Alias-based baseline (`alias` field)

By default (no alias given), the automatic baseline cell (when prompt_id is set) resolves to the prompt's production alias if one exists, otherwise its latest committed version. Passing alias points it at a different named alias's version instead — verified here against a prompt whose production alias is pinned to v1 while a staging alias was separately promoted to v2, requesting v2 explicitly plus alias: "staging":

curl -X POST $ACRUXCORE_BASE_URL/experiments \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1", "prompt_id": "07494047-7380-4324-acf9-069a40265323", "name": "alias baseline demo", "version_ids": ["fd5da811-2fc8-4df0-a6b8-8198dff7a678"], "models": ["gpt-4o-mini"], "judge_model": "gpt-4o-mini", "alias": "staging"}'

Response (status 201) — alias is stored verbatim on config, alongside models/judgeModel/versionIds:

{
  "id": "084f43d1-21f5-4235-b55e-7ebfc0946b76",
  "teamId": "6b319ddb-bae8-469e-9af1-ffac0d4df846",
  "datasetId": "20e0308d-ec9c-40ac-8cd9-c20b4bb95df1",
  "promptId": "07494047-7380-4324-acf9-069a40265323",
  "name": "alias baseline demo",
  "config": { "alias": "staging", "models": ["gpt-4o-mini"], "judgeModel": "gpt-4o-mini", "versionIds": ["fd5da811-2fc8-4df0-a6b8-8198dff7a678"] },
  "createdBy": "9bf448bd-5c2f-43d0-be81-08b117195b8d",
  "createdAt": "2026-10-13T13:06:44.769Z",
  "runs": []
}

alias is only consulted when POST /experiments/:id/runs resolves the automatic baseline cell (AliasesService.resolveBaselineVersion) — it has no effect on the explicit version_ids grid itself. An alias that doesn't exist on this prompt fails the request synchronously: RunsService .startRun awaits resolveGrid (which calls resolveBaselineVersion) before creating the run row, so the NotFoundError it throws propagates straight out of POST /experiments/:id/runs as a synchronous 404 — no run is ever created, and there is nothing to look up afterwards on GET /runs/:id.

Prompt-mismatch warning (`promptMismatchWarning`, response only)

Not curl-verified in this environment: reproducing it needs a dataset whose examples were built via POST /datasets/from-feedback, which requires a real POST /gateway/chat/completions call against a configured LLM provider to populate a trace's sourcePromptVersionId — no provider key (OPENAI_TEST_KEY/ANTHROPIC_TEST_KEY) is configured in this environment. Documented from source instead (experiments.service.ts/experiments.types.ts on this branch): when the dataset's examples were sourced from a prompt other than prompt_id, the 201 response carries an extra top-level promptMismatchWarning object (camelCase, unlike the optimize endpoint's snake_case prompt_mismatch_warning — this endpoint's whole response is camelCase):

{
  "id": "9462c9b7-62a7-4f01-80fe-71f142cdd02b",
  "teamId": "c2e28d00-c7ed-4c27-85d9-434927282f67",
  "datasetId": "4595c340-d7fa-4d76-977e-76b60ebefb62",
  "promptId": "173ad5cd-969c-4a04-8896-fbf625bbc2e5",
  "name": "E3 Task7 greeting sweep",
  "config": { "models": ["gpt-4o-mini-e3t7"], "judgeModel": "gpt-4o-mini-e3t7", "versionIds": ["b5327b3e-3c45-4858-ac62-1e5852279624"] },
  "createdBy": "6937e9cb-6754-4f45-aac8-a780d2e3ef57",
  "createdAt": "2026-07-07T14:26:50.940Z",
  "runs": [],
  "promptMismatchWarning": {
    "mismatchedPrompts": [
      { "promptId": "1e5e7cd3-7570-402c-97ff-e5b0c4eadfb3", "name": "support-reply", "exampleCount": 2 }
    ]
  }
}

mismatchedPrompts groups by the other prompt each mismatched example was sourced from, with exampleCount counting only that prompt's examples in this dataset. It's informational only — the experiment is created either way — and the field is omitted entirely (not null) when every example either matches prompt_id or carries no resolvable lineage (manually-added examples, or a since-deleted source version). It is only computed when prompt_id is given, since without a target prompt there is nothing to compare an example's lineage against.

POST /api/v1/experiments/:id/runs

Starts a run: freezes the dataset's current examples, resolves the grid (here: the explicit v2 cell + an automatic production-baseline cell, since production still pointed at v1 — not among the explicit version_ids), and enqueues a BullMQ Flow. Returns immediately — the actual gateway calls happen asynchronously in the worker process.

curl
TypeScript
Python

curl -X POST $ACRUXCORE_BASE_URL/experiments/9462c9b7-62a7-4f01-80fe-71f142cdd02b/runs \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY"

const run = await hub.runs.start({
  experimentId: '9462c9b7-62a7-4f01-80fe-71f142cdd02b',
});

run = await hub.experiments.start_run(
    '9462c9b7-62a7-4f01-80fe-71f142cdd02b',
)

Response (status 202):

{ "run_id": "891710e4-1ac4-4bb7-b618-0f81e31aa291", "status": "queued" }

Error responses

Nonexistent experiment id (status 404):

curl -X POST $ACRUXCORE_BASE_URL/experiments/00000000-0000-0000-0000-000000000000/runs \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY"

{ "error": { "code": "NOT_FOUND", "message": "Experiment not found." } }

An experiment saved before judges were chosen per experiment has no config.judgeModel (status 400). The API does not guess one, and no run row is created. Create a new experiment with a judge_model. The same check re-runs the registry lookup, so an experiment whose judge model has since been removed from Gateway → Models fails here too:

curl -X POST $ACRUXCORE_BASE_URL/experiments/0a9c535d-0ae1-482a-93c2-14fe906b7029/runs \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY"

{ "error": { "code": "VALIDATION_ERROR", "message": "This experiment has no judge model — it was created before judges were chosen per experiment. Create a new experiment with a judge_model." } }

An experiment whose judge_prompt_id was deleted after it was created (status 400). The prompt is checked again here, before a run row or any model call exists:

curl -X POST $ACRUXCORE_BASE_URL/experiments/<experiment-id>/runs \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

{ "error": { "code": "VALIDATION_ERROR", "message": "judge_prompt_id does not refer to a prompt in this team." } }

GET /api/v1/runs

Lists the team's runs, newest first — the run history. Each row carries the dataset and prompt the run evaluated, the shape of its frozen grid, and the scores it produced, so you can find the run worth opening without fetching each one's report.

kind is "optimize" for a run started by POST /prompts/:id/optimize, "local" for a run your SDK ran, and "evaluation" otherwise. The kind is set when the run is created, so a queued optimize run already reads as "optimize". avgScore, passRate and topVariantLabel are null — never 0 — until the run has at least one judge-scored result, so an unscored run is never mistaken for one that scored badly. avgScore is the mean across every scored result in the run, weighted by example (a 1-example variant does not count the same as a 100-example one), and topVariantLabel names the highest-mean variant.

Query params (all optional):

Param	Type	Default	Meaning
`status`	`queued` \| `running` \| `succeeded` \| `failed`	—	Lifecycle filter
`dataset_id`	uuid	—	Only runs whose experiment used this dataset
`prompt_id`	uuid	—	Only runs whose experiment named this prompt
`page`	integer ≥ 1	`1`	1-based page number
`limit`	integer 1–100	`20`	Rows per page

curl
TypeScript
Python

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs

const runs = await hub.runs.list();

runs = await hub.runs.list()

Response (status 200) — one optimize run and one evaluation run, both finished:

{
  "data": [
    {
      "id": "df727319-f64d-4ebd-8e3d-fa1b43d0f377",
      "status": "succeeded",
      "kind": "optimize",
      "experimentId": "fc90fd51-4f00-400f-b547-8a17017434dd",
      "experimentName": "optimize",
      "datasetId": "d95a4068-3415-48fe-959a-ba7cfd3d96f5",
      "datasetName": "support replies",
      "promptId": "1e5e7cd3-7570-402c-97ff-e5b0c4eadfb3",
      "promptName": "support-reply",
      "variantCount": 3,
      "modelCount": 1,
      "exampleCount": 3,
      "results": { "total": 9, "succeeded": 9, "errored": 0, "scored": 9 },
      "avgScore": 83.9,
      "passRate": 1,
      "topVariantLabel": "candidate-A",
      "startedBy": { "id": "035f4fca-ec77-47e2-b6de-755b0111c798", "name": "Run History", "email": "runhist@example.com" },
      "createdAt": "2026-08-04T03:32:57.141Z",
      "startedAt": "2026-08-04T03:33:00.093Z",
      "endedAt": "2026-08-04T03:33:08.434Z",
      "durationMs": 8341
    },
    {
      "id": "54a06825-dce7-4c53-9c4a-7115ebd91ace",
      "status": "succeeded",
      "kind": "evaluation",
      "experimentId": "01009014-6f50-41a8-9089-ddcc8aca75e9",
      "experimentName": "support reply sweep",
      "datasetId": "d95a4068-3415-48fe-959a-ba7cfd3d96f5",
      "datasetName": "support replies",
      "promptId": "1e5e7cd3-7570-402c-97ff-e5b0c4eadfb3",
      "promptName": "support-reply",
      "variantCount": 2,
      "modelCount": 1,
      "exampleCount": 3,
      "results": { "total": 6, "succeeded": 6, "errored": 0, "scored": 6 },
      "avgScore": 81.7,
      "passRate": 1,
      "topVariantLabel": "production",
      "startedBy": { "id": "035f4fca-ec77-47e2-b6de-755b0111c798", "name": "Run History", "email": "runhist@example.com" },
      "createdAt": "2026-08-04T03:32:35.362Z",
      "startedAt": "2026-08-04T03:32:35.398Z",
      "endedAt": "2026-08-04T03:32:43.461Z",
      "durationMs": 8063
    }
  ],
  "total": 2,
  "page": 1,
  "limit": 20
}

A run still in flight lists the partial numbers it has produced so far rather than nothing — polled while the run above was mid-flight, 4 of its 6 results were already scored:

{ "status": "running", "results": { "total": 6, "succeeded": 6, "errored": 0, "scored": 4 }, "avgScore": 82.5, "durationMs": null }

Paging (total is the full filtered count, not the page's length):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs?limit=1&page=2"

{ "total": 2, "page": 2, "limit": 1 }

A filter that matches nothing returns an empty page, not a 404:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs?status=failed"

{ "data": [], "total": 0, "page": 1, "limit": 20 }

Error responses

A status outside the four lifecycle values (status 400):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs?status=cancelled"

{ "error": { "code": "VALIDATION_ERROR", "message": "Invalid enum value. Expected 'queued' | 'running' | 'succeeded' | 'failed', received 'cancelled'" } }

A limit above the 100 ceiling (status 400):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs?limit=500"

{ "error": { "code": "VALIDATION_ERROR", "message": "Number must be less than or equal to 100" } }

GET /api/v1/runs/:id

Polled immediately after starting the run above — the real worker had not yet picked up any of the 4 cell jobs (2 grid cells × 2 dataset examples):

curl
TypeScript
Python

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/891710e4-1ac4-4bb7-b618-0f81e31aa291

const run = await hub.runs.get('891710e4-1ac4-4bb7-b618-0f81e31aa291');

run = await hub.runs.get('891710e4-1ac4-4bb7-b618-0f81e31aa291')

Response (status 200) — in-flight, queued, zero results so far:

{
  "id": "891710e4-1ac4-4bb7-b618-0f81e31aa291",
  "experimentId": "9462c9b7-62a7-4f01-80fe-71f142cdd02b",
  "status": "queued",
  "startedAt": null,
  "endedAt": null,
  "error": null,
  "createdAt": "2026-07-07T14:27:16.920Z",
  "grid": [
    { "model": "gpt-4o-mini-e3t7", "cellKey": "v2|gpt-4o-mini-e3t7", "variantKind": "version", "variantLabel": "v2", "promptVersionId": "b5327b3e-3c45-4858-ac62-1e5852279624", "isProductionBaseline": false },
    { "model": "gpt-4o-mini-e3t7", "cellKey": "production|gpt-4o-mini-e3t7", "variantKind": "version", "variantLabel": "production", "promptVersionId": "8250ce7b-cc5f-463b-83be-3a497f3f6f0a", "isProductionBaseline": true }
  ],
  "exampleCount": 2,
  "results": { "total": 0, "succeeded": 0, "errored": 0 }
}

Polled again ~2 seconds later, same run id — the real worker process had by then dequeued and processed all 4 cell jobs (2 grid cells x 2 examples) against the real OpenAI account, and the finalize job marked it succeeded:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/891710e4-1ac4-4bb7-b618-0f81e31aa291

Response (status 200) — succeeded, all 4 results in:

{
  "id": "891710e4-1ac4-4bb7-b618-0f81e31aa291",
  "experimentId": "9462c9b7-62a7-4f01-80fe-71f142cdd02b",
  "status": "succeeded",
  "startedAt": null,
  "endedAt": "2026-07-07T14:27:18.799Z",
  "error": null,
  "createdAt": "2026-07-07T14:27:16.920Z",
  "grid": [
    { "model": "gpt-4o-mini-e3t7", "cellKey": "v2|gpt-4o-mini-e3t7", "variantKind": "version", "variantLabel": "v2", "promptVersionId": "b5327b3e-3c45-4858-ac62-1e5852279624", "isProductionBaseline": false },
    { "model": "gpt-4o-mini-e3t7", "cellKey": "production|gpt-4o-mini-e3t7", "variantKind": "version", "variantLabel": "production", "promptVersionId": "8250ce7b-cc5f-463b-83be-3a497f3f6f0a", "isProductionBaseline": true }
  ],
  "exampleCount": 2,
  "results": { "total": 4, "succeeded": 4, "errored": 0 }
}

Note: startedAt stayed null through this observed run — the run engine does not appear to stamp it on transition to succeeded in this build; only endedAt is populated. Documented as observed, not as a defect judgment.

Drift note (re-verified 2026-07-12): isProductionBaseline on each grid entry above was missing from this doc's original example even though the live server includes it for runs started via POST /experiments/:id/runs — added back here to match observed behavior. By contrast, a run started via the optimize loop (POST /prompts/:promptId/optimize, documented in optimize.mdx) does NOT include isProductionBaseline on its grid entries (only GET /runs/:id/report's variants/cells carry that field for an optimize-originated run) — a real, observed difference between the two run-creation code paths, not a doc error.

Each of the 4 result rows landed in eval_results with real, distinct model output and its own real gateway trace_id (verified via a read-only SELECT, not part of the HTTP surface but confirming the worker really called the gateway rather than stubbing it):

v2 / Alice          -> "Hey Alice! 🌟 It's a fantastic day and I'm so excited to see you shine! 🎉"
v2 / Bob            -> "Hey there, Bob! 🎉 Get ready to seize the day and make it amazing!"
production / Alice  -> "Hello, Alice! It's wonderful to see you!"
production / Bob    -> "Hello, Bob! It's great to see you!"

Error responses

Nonexistent run id (status 404):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/00000000-0000-0000-0000-000000000000

{ "error": { "code": "NOT_FOUND", "message": "Run not found." } }

GET /api/v1/runs/:id/report

Comparison report (Phase 5 E5): the full (variant x model) matrix with per-cell averages, each non-baseline cell's regression delta vs. the same-model production-baseline cell, a leaderboard, and an advisory winner. Computed on read from the run's eval_result rows — no dedicated report table. Verified against a fresh experiment (prompt "support-reply", v1 promoted to production with a vague, unconstrained system message; v2 an explicit "reply in exactly two sentences, no exclamation marks" instruction), run against 2 dataset examples whose criteria both demand the same strict two-sentence format — arranged specifically so v2 and production diverge sharply on the judge's score, rather than landing on a flat/unknown delta that wouldn't demonstrate the feature.

Note: the judge model is the experiment's own judge_model, chosen when the experiment is created — distinct from whatever public model name(s) the experiment's grid sweeps. It must be registered via POST /gateway/models like any other model; POST /experiments rejects a name that is not.

curl
TypeScript
Python

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/6e8bcaaf-8b5b-40f5-9b59-6d83eec71596/report

const report = await hub.runs.getReport('6e8bcaaf-8b5b-40f5-9b59-6d83eec71596');

report = await hub.runs.get_report('6e8bcaaf-8b5b-40f5-9b59-6d83eec71596')

Response (status 200) — v2 clearly beats the production baseline: avgScore 85 vs 20, passRate 1 vs 0, deltaVsBaseline.label "improved" (+65 score points), and the winner is v2:

{
  "runId": "6e8bcaaf-8b5b-40f5-9b59-6d83eec71596",
  "status": "succeeded",
  "models": ["gpt-4o-mini-e5t3"],
  "variants": [
    { "variantKind": "version", "promptVersionId": "3bbf603b-e99e-475a-adf6-ab9edcb0ecc0", "variantLabel": "v2", "isProductionBaseline": false },
    { "variantKind": "version", "promptVersionId": "2bd8f192-d4cf-4123-9816-4d8eba9158bb", "variantLabel": "production", "isProductionBaseline": true }
  ],
  "cells": [
    {
      "cellKey": "v2|gpt-4o-mini-e5t3",
      "variantLabel": "v2",
      "model": "gpt-4o-mini-e5t3",
      "isProductionBaseline": false,
      "avgScore": 85,
      "passRate": 1,
      "exampleCount": 2,
      "scoredCount": 2,
      "unscoredCount": 0,
      "deltaVsBaseline": { "score": 65, "passRate": 1, "label": "improved" }
    },
    {
      "cellKey": "production|gpt-4o-mini-e5t3",
      "variantLabel": "production",
      "model": "gpt-4o-mini-e5t3",
      "isProductionBaseline": true,
      "avgScore": 20,
      "passRate": 0,
      "exampleCount": 2,
      "scoredCount": 2,
      "unscoredCount": 0,
      "deltaVsBaseline": null
    }
  ],
  "leaderboard": ["v2|gpt-4o-mini-e5t3", "production|gpt-4o-mini-e5t3"],
  "winner": { "cellKey": "v2|gpt-4o-mini-e5t3", "variantLabel": "v2", "model": "gpt-4o-mini-e5t3", "avgScore": 85 }
}

Which judge graded the run

judge names the judge model and, when the experiment used a custom judge prompt, the prompt version fixed at run start. Both are null for runs created before this field existed; promptVersionId is null when the built-in judge graded the run. Verified with a run graded by a custom judge prompt:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/406968cd-9492-4967-b6f0-f7522d3fcbb9/report

Response (status 200):

{
  "runId": "406968cd-9492-4967-b6f0-f7522d3fcbb9",
  "status": "succeeded",
  "models": [
    "gpt-4o-mini-or"
  ],
  "variants": [
    {
      "variantKind": "version",
      "promptVersionId": "6059a265-eaff-4107-b98a-bf4808e630e5",
      "variantLabel": "v2",
      "isProductionBaseline": true
    }
  ],
  "cells": [
    {
      "cellKey": "v2|gpt-4o-mini-or",
      "variantLabel": "v2",
      "model": "gpt-4o-mini-or",
      "isProductionBaseline": true,
      "avgScore": 16.666666666666668,
      "passRate": 0,
      "exampleCount": 3,
      "scoredCount": 3,
      "unscoredCount": 0,
      "deltaVsBaseline": null
    }
  ],
  "leaderboard": [
    "v2|gpt-4o-mini-or"
  ],
  "winner": {
    "cellKey": "v2|gpt-4o-mini-or",
    "variantLabel": "v2",
    "model": "gpt-4o-mini-or",
    "avgScore": 16.666666666666668
  },
  "judge": {
    "model": "gpt-4o-mini-or",
    "promptVersionId": "66ae6460-f8f8-4140-aa2b-5baef81c8fd3"
  }
}

Cells with failed or unscored examples

Each cell also carries erroredCount, the examples whose candidate call failed so no answer exists, and unscoredReason, the most common reason its examples have no score together with how many examples share it. The reason is the example's errorMessage when the candidate call failed, otherwise the judge's reason. unscoredReason is null when every example in the cell was scored. A run still ends succeeded unless every example errored, and avgScore averages only the scored examples, so these two fields are where a partial failure shows.

The report also has a top-level unscoredReason, counted over every result in the run. A reason that comes second in every cell can still be the most common one in the run, so this field is not built from the cells' own reasons. When two reasons have the same count, the one seen first wins. It is null when every result was scored.

Verified with a run of one prompt version on two models, where revoked-mini and the judge model both use a connection whose key the provider rejects:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/ebe191f6-4e58-4a67-8208-bf8bd9a1ac02/report

Response (status 200):

{
    "runId": "ebe191f6-4e58-4a67-8208-bf8bd9a1ac02",
    "status": "succeeded",
    "models": [
        "gpt-4o-mini",
        "revoked-mini"
    ],
    "variants": [
        {
            "variantKind": "version",
            "promptVersionId": "f0d6ec0d-4de6-4b23-9a4e-056011767fd5",
            "variantLabel": "v1",
            "versionNumber": 1,
            "isProductionBaseline": true
        }
    ],
    "cells": [
        {
            "cellKey": "v1|gpt-4o-mini",
            "variantLabel": "v1",
            "model": "gpt-4o-mini",
            "isProductionBaseline": true,
            "avgScore": null,
            "passRate": null,
            "exampleCount": 3,
            "scoredCount": 0,
            "unscoredCount": 3,
            "erroredCount": 0,
            "unscoredReason": {
                "message": "judge call failed: Provider error (401): OpenAI request failed with status 401",
                "count": 3
            },
            "deltaVsBaseline": null,
            "namedScores": []
        },
        {
            "cellKey": "v1|revoked-mini",
            "variantLabel": "v1",
            "model": "revoked-mini",
            "isProductionBaseline": true,
            "avgScore": null,
            "passRate": null,
            "exampleCount": 3,
            "scoredCount": 0,
            "unscoredCount": 3,
            "erroredCount": 3,
            "unscoredReason": {
                "message": "Provider error (401): OpenAI request failed with status 401",
                "count": 3
            },
            "deltaVsBaseline": null,
            "namedScores": []
        }
    ],
    "leaderboard": [
        "v1|gpt-4o-mini",
        "v1|revoked-mini"
    ],
    "winner": null,
    "unscoredReason": {
        "message": "judge call failed: Provider error (401): OpenAI request failed with status 401",
        "count": 3
    },
    "judge": {
        "model": "revoked-mini",
        "promptVersionId": null
    }
}

Error responses

Nonexistent run id (status 404) — same message/code as GET /runs/:id:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/00000000-0000-0000-0000-000000000000/report

{ "error": { "code": "NOT_FOUND", "message": "Run not found." } }

Which version is behind each variant

Each entry in variants carries versionNumber, the number of the prompt version that variant ran. A baseline is labelled by its alias (production); versionNumber says which version that alias pointed to when the run started, so the dashboard shows it as "v1 (production)". It is null for an optimizer candidate or a version that was deleted.

curl $ACRUXCORE_BASE_URL/runs/9666961b-d72c-415f-80ea-e13d99b77977/report \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY"

Response (status 200), variants only:

"variants": [
  { "variantKind": "version", "promptVersionId": "660b027a-8ff6-4ee6-8f2e-4b913109dba3", "variantLabel": "v2", "versionNumber": 2, "isProductionBaseline": false },
  { "variantKind": "version", "promptVersionId": "601f5f1c-7f54-47f9-ab8f-ef4387a0db53", "variantLabel": "production", "versionNumber": 1, "isProductionBaseline": true }
]

GET /api/v1/runs/:id/cells/:cellKey

Drill-down (Phase 5 E5): one grid cell's per-example outputs, judge reasoning, and traces. cellKey is ${variantLabel}|${model} (as minted into the run's grid) and must be URL-encoded (| -> %7C) — decoded again by Express before it reaches the controller. Each example in the response also carries history: the same array frozen onto the dataset example at run-start time (see history on GET /api/v1/datasets/:id in the datasets reference), or null for a single-turn example — this is the exact conversation that was replayed ahead of input for that cell's completion, not the live (possibly since-edited) dataset value. Not separately re-curled below: it's the same frozen JSON already curl-verified on the dataset endpoint, and the run/judge wiring is covered by the real-Postgres integration test apps/api/src/evaluations/runs/run-engine.test.ts. Curled against the same run as the report above, both of its two cells:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs/6e8bcaaf-8b5b-40f5-9b59-6d83eec71596/cells/v2%7Cgpt-4o-mini-e5t3"

Response (status 200) — v2's two examples: one perfect (score 100), one partial-credit (score 70, missing the concrete next step) — both passed:

{
  "cellKey": "v2|gpt-4o-mini-e5t3",
  "variantLabel": "v2",
  "model": "gpt-4o-mini-e5t3",
  "examples": [
    {
      "exampleId": "95f4e926-c036-4fa6-881a-4453b7c3d3fd",
      "input": { "complaint": "My package arrived three days late and the box was crushed." },
      "criteria": "Response must be EXACTLY two sentences: first sentence apologizes and names the specific issue (late delivery / crushed box), second sentence states one concrete next step (e.g. replacement or refund). No exclamation marks. No greeting or sign-off.",
      "output": "I apologize for the delay and the condition of your package. Please provide your order number so we can initiate a replacement or refund for you.",
      "errorMessage": null,
      "score": 100,
      "passed": true,
      "reason": "The output consists of exactly two sentences: the first sentence apologizes for the delay and mentions the condition of the package, while the second sentence clearly states the next step of providing the order number for a replacement or refund.",
      "traceId": "04e58fce-3fd0-4488-907e-d0a79a7cc47d",
      "judgeTraceId": "2fd53824-36bd-48f9-9550-b9ba43ae1663"
    },
    {
      "exampleId": "e4db8a4f-f530-49ac-87e7-f4f208b2c19b",
      "input": { "complaint": "I was charged twice for the same order and nobody has responded to my emails." },
      "criteria": "Response must be EXACTLY two sentences: first sentence apologizes and names the specific issue (double charge / no response to emails), second sentence states one concrete next step (e.g. refund of duplicate charge). No exclamation marks. No greeting or sign-off.",
      "output": "I apologize for the inconvenience caused by the double charge and the lack of response to your emails. Please provide your order number so that I can investigate this issue further.",
      "errorMessage": null,
      "score": 70,
      "passed": true,
      "reason": "The output contains an apology for the double charge and the lack of response to emails, fulfilling the first sentence requirement. However, it does not provide a concrete next step, which is necessary to fully meet the criteria.",
      "traceId": "2b8607f6-d7e8-43bb-92ff-3c105efb7ed4",
      "judgeTraceId": "db82535f-3c88-478f-8ff2-489a8c2e5bb8"
    }
  ]
}

The production-baseline cell, for contrast — both examples score 20, passed: false, and the judge's reason names the exact violation (too many sentences, greeting/sign-off present):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs/6e8bcaaf-8b5b-40f5-9b59-6d83eec71596/cells/production%7Cgpt-4o-mini-e5t3"

Response (status 200) — excerpted to one example (the other is the same shape):

{
  "cellKey": "production|gpt-4o-mini-e5t3",
  "variantLabel": "production",
  "model": "gpt-4o-mini-e5t3",
  "examples": [
    {
      "exampleId": "e4db8a4f-f530-49ac-87e7-f4f208b2c19b",
      "input": { "complaint": "I was charged twice for the same order and nobody has responded to my emails." },
      "criteria": "Response must be EXACTLY two sentences: first sentence apologizes and names the specific issue (double charge / no response to emails), second sentence states one concrete next step (e.g. refund of duplicate charge). No exclamation marks. No greeting or sign-off.",
      "output": "I'm really sorry to hear about the double charge and the delay in getting a response. We certainly want to resolve this issue for you as quickly as possible. \n\nTo assist you better, could you please provide me with the following details? \n\n1. The order number.\n2. The date of the order.\n3. Any reference numbers or transaction details for both charges.\n\nOnce I have that information, I'll be able to look into it right away and help you sort this out. Thank you for your patience, and I appreciate your understanding!",
      "errorMessage": null,
      "score": 20,
      "passed": false,
      "reason": "The output does not meet the requirement of being exactly two sentences; it contains multiple sentences and includes a greeting and sign-off, which are not allowed.",
      "traceId": "c43822d7-5562-405e-8fa0-02a7acb3995a",
      "judgeTraceId": "3e396879-160e-4f3d-9f05-95dde37ece79"
    }
  ]
}

A cell that failed

errorMessage carries why a cell produced nothing, and is null on every cell that ran. It is not the same field as reason: reason is the judge's verdict, and stays null when there was no output to judge. A failed cell has output: null, traceId: null and no judge fields at all, so errorMessage is the only account of what happened.

Curled against a run started against a model the team had not registered:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs/e9a53392-bf87-47ee-90f9-361e65f19cbc/cells/v1%7Cgpt-4o-mini"

Response (status 200) — excerpted to one of its two examples (the other is the same shape):

{
  "cellKey": "v1|gpt-4o-mini",
  "variantLabel": "v1",
  "model": "gpt-4o-mini",
  "examples": [
    {
      "exampleId": "3ef6fb9a-3cce-4809-a5f8-b0136f34b738",
      "input": { "question": "What is the capital of France?" },
      "criteria": "The answer must be correct.",
      "history": null,
      "output": null,
      "errorMessage": "Model 'gpt-4o-mini' is not registered. Add it under Gateway → Models.",
      "score": null,
      "passed": null,
      "reason": null,
      "traceId": null,
      "judgeTraceId": null
    }
  ]
}

The same run's own row says it too — error on GET /runs/:id names the shared reason once when every cell failed the same way, and the first one when they differ:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs/e9a53392-bf87-47ee-90f9-361e65f19cbc"

Response (status 200), excerpted to the three fields this section is about — id, experimentId, grid, exampleCount and the timestamps are unchanged and shown under GET /api/v1/runs/:id above:

{
  "status": "failed",
  "error": "Every cell failed. Model 'gpt-4o-mini' is not registered. Add it under Gateway → Models.",
  "results": { "total": 2, "succeeded": 0, "errored": 2 }
}

Error responses

Nonexistent run id (status 404) — same message/code as GET /runs/:id/report:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs/00000000-0000-0000-0000-000000000000/cells/v2%7Cgpt-4o-mini-e5t3"

{ "error": { "code": "NOT_FOUND", "message": "Run not found." } }

A cellKey that does not match any of the run's grid cells (status 200, not an error) — the run itself is real, the cell key is simply not one of its grid entries, so it returns gracefully with an empty examples array:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" "$ACRUXCORE_BASE_URL/runs/6e8bcaaf-8b5b-40f5-9b59-6d83eec71596/cells/nonexistent%7Cmodel-x"

{ "cellKey": "nonexistent|model-x", "variantLabel": "nonexistent", "model": "model-x", "examples": [] }

GET /api/v1/runs/:id/compare

Two grid cells side by side, one row per dataset example. A cell's average can stay the same while it fixes some examples and breaks others; this read shows which ones.

Query params:

base_cell (required) — a cell key in the run named by :id, URL-encoded (| → %7C).
target_cell (required) — a cell key in the target run.
target_run_id — the run holding the target cell. Omit it to compare two cells of one run.

Each example's change is decided on pass/fail: improved (fail → pass), regressed (pass → fail), unchanged, unscored (a side has no verdict), or missing (only one side has a result). scoreDelta is target.score - base.score and never decides change. counts holds one number per change, and they add up to the number of examples.

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  "$ACRUXCORE_BASE_URL/runs/a2c402ba-9e4a-4382-8942-2d759b5f1378/compare?base_cell=production%7Cgpt-4o-mini&target_cell=v2%7Cgpt-4o-mini"

Response (status 200):

{
  "base": {
    "runId": "a2c402ba-9e4a-4382-8942-2d759b5f1378",
    "cellKey": "production|gpt-4o-mini",
    "variantLabel": "production",
    "model": "gpt-4o-mini"
  },
  "target": {
    "runId": "a2c402ba-9e4a-4382-8942-2d759b5f1378",
    "cellKey": "v2|gpt-4o-mini",
    "variantLabel": "v2",
    "model": "gpt-4o-mini"
  },
  "counts": {
    "improved": 2,
    "regressed": 1,
    "unchanged": 0,
    "unscored": 0,
    "missing": 0
  },
  "examples": [
    {
      "exampleId": "dd9e6463-1909-4819-bb36-90ac68a57505",
      "input": {
        "message": "Any idea when SSO is landing? We have been waiting since March and our security review is blocked on it."
      },
      "criteria": "Say whether we have a date. If we do not, say so plainly, say what we do know, and tell them how they will hear when it changes. Four sentences at most, and do not promise a month we have not committed to.",
      "base": {
        "output": "Hello,\n\nThank you for reaching out and for your patience regarding the Single Sign-On (SSO) feature. I understand how important this is for your security review process.\n\nAs of my last update, our team has been diligently working on implementing SSO, and I can confirm that it is a priority for us. While I do not have a specific release date at the moment, I recommend keeping an eye on our official communication channels for updates. \n\nIn the meantime, if you have any other questions or if there\u2019s anything else I can assist you with, please feel free to let me know.\n\nThank you for your understanding!\n\nBest regards,\n[Your Name]  \nAcrux Core Support Team",
        "errorMessage": null,
        "score": 70,
        "passed": true,
        "reason": "The output acknowledges the lack of a specific release date for the SSO feature and suggests monitoring official communication channels for updates. However, it could be more concise and directly state the absence of a date in the first sentence, as well as provide a clearer next step with a time frame.",
        "traceId": "6b79ce66-d667-4fd9-946b-85579fd1ac75"
      },
      "target": {
        "output": "You've mentioned that you're waiting for the SSO feature, which is currently holding up your security review. I understand how important this is for your project. I recommend checking back with us in two weeks for an update on the SSO implementation timeline.",
        "errorMessage": null,
        "score": 60,
        "passed": false,
        "reason": "The output acknowledges the user's concern but does not clearly state whether a date for the SSO feature is available. It lacks a direct apology and does not provide a concrete next step with a specific time frame, which is essential according to the feedback.",
        "traceId": "51b56fa2-fa87-4e14-a1cb-b7d29b98e482"
      },
      "scoreDelta": -10,
      "change": "regressed"
    },
    {
      "exampleId": "c3c69779-d6db-48f0-a504-7d181328335f",
      "input": {
        "message": "I was charged for another month yesterday even though I cancelled. Your help page says refunds only inside 14 days and I am on day 16. Is there really nothing you can do?"
      },
      "criteria": "Say plainly that day 16 is outside the 14-day window, offer what we can do instead, and end with one concrete next step. Four sentences at most, and at most one apology.",
      "base": {
        "output": "Dear [Customer's Name],\n\nThank you for reaching out to us regarding your recent charge. I understand how frustrating this situation must be, especially after you took the steps to cancel your subscription.\n\nWhile our policy states that refunds are typically processed within 14 days of the charge, I would be happy to look into your case further. Could you please provide me with the email address associated with your account or any relevant details about your cancellation? This will help me to investigate the matter and see if there are any options available for you.\n\nThank you for your patience, and I look forward to assisting you.\n\nBest regards,\n\n[Your Name]  \nCustomer Support Agent  \nAcrux Core",
        "errorMessage": null,
        "score": 40,
        "passed": false,
        "reason": "The output does not clearly state that day 16 is outside the 14-day refund window, and it does not provide a concrete next step or time frame. It also exceeds the four-sentence limit.",
        "traceId": "8e468ee6-8e92-4b6f-a0d7-ad19549b4741"
      },
      "target": {
        "output": "You are experiencing a charge for your subscription despite having canceled, and you are outside the 14-day refund window. I understand how frustrating this situation can be. I can escalate your case to our billing department for further review and they will get back to you within 3 business days to provide an update.",
        "errorMessage": null,
        "score": 70,
        "passed": true,
        "reason": "The output identifies the issue of being outside the 14-day refund window and includes an apology. It offers a next step by stating that the case can be escalated to the billing department, which provides a concrete action. However, it could be more direct in stating that day 16 is outside the refund window.",
        "traceId": "f5fba8af-18f7-42ad-bdfe-14e32f78e722"
      },
      "scoreDelta": 30,
      "change": "improved"
    },
    {
      "exampleId": "07885ab9-6bca-4eec-bee0-eb0a6f818300",
      "input": {
        "message": "Our API keys stopped working this morning \u2014 every call comes back 401 and nothing changed on our side. We have a customer demo at 2pm and I am getting nervous."
      },
      "criteria": "Name the 401 errors in the first sentence, give them one thing to try right now, and say when we will come back to them. There is a demo in a few hours, so keep it to four sentences.",
      "base": {
        "output": "Hello,\n\nI understand how frustrating this situation can be, especially with a demo coming up soon. A 401 error typically indicates that the authentication has failed, which could happen for several reasons. Here are a few steps you can take to troubleshoot the issue:\n\n1. **Check API Key**: Ensure that the API key you\u2019re using is correct. Double-check for any typos or extra spaces.\n\n2. **Re-Generate API Key**: If you believe the key might have been compromised or changed, consider regenerating a new API key in your dashboard.\n\n3. **Scope and Permissions**: Verify if there have been any changes in the permissions or scope assigned to the API key you are using.\n\n4. **Rate Limits**: Ensure that you have not exceeded any rate limits that might restrict access.\n\n5. **Backend Changes**: Although you mentioned that nothing changed on your side, it may be worth checking if there were any updates or changes on our end that could have affected the API access.\n\nTo assist you further, could you please provide the following information?\n- The specific API endpoints you are trying to access\n- The way you are making the calls (e.g., tool, method)\n- Any recent changes to your application that could be relevant\n\nWe\u2019re here to help you resolve this issue as quickly as possible. \n\nBest regards,  \n[Your Name]  \nAcrux Core Support Team",
        "errorMessage": null,
        "score": 30,
        "passed": false,
        "reason": "The output does not name the 401 error in the first sentence, does not apologize, and exceeds four sentences. It also lacks a concrete next step and a time frame for follow-up.",
        "traceId": "2b7bb00b-4976-4dc8-9981-f5026ef7ffcf"
      },
      "target": {
        "output": "You've experienced a sudden issue with your API keys returning a 401 error, even though no changes were made on your side. I understand how urgent this is, especially with your demo at 2pm. Please verify if there are any recent changes in your account settings or if your API keys have been regenerated. Meanwhile, I'll have our technical team review your API keys and provide feedback within the next hour.",
        "errorMessage": null,
        "score": 70,
        "passed": true,
        "reason": "The output identifies the 401 error in the first sentence and acknowledges the urgency of the situation. It provides a suggestion to verify account settings or regenerated API keys, which serves as a concrete next step. However, it could be more concise, as it slightly exceeds the four-sentence limit.",
        "traceId": "26242b8e-08d6-4746-bedd-c1905d681527"
      },
      "scoreDelta": 40,
      "change": "improved"
    }
  ]
}

The two runs must have graded the same dataset, because examples are paired by dataset example id. GET /api/v1/runs/:id returns datasetId so a client can find runs to compare against:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/runs/a2c402ba-9e4a-4382-8942-2d759b5f1378

Response (status 200):

{
  "id": "a2c402ba-9e4a-4382-8942-2d759b5f1378",
  "experimentId": "9f9d6b8c-145a-4cf7-8993-c1023ec24a0d",
  "datasetId": "c776580d-e46e-45a4-8847-b00004871502",
  "status": "succeeded",
  "startedAt": "2026-09-06T11:24:41.342Z",
  "endedAt": "2026-09-06T11:24:50.360Z",
  "error": null,
  "createdAt": "2026-09-06T11:24:41.327Z",
  "grid": [
    {
      "model": "gpt-4o-mini",
      "cellKey": "v2|gpt-4o-mini",
      "variantKind": "version",
      "variantLabel": "v2",
      "promptVersionId": "6059a265-eaff-4107-b98a-bf4808e630e5",
      "isProductionBaseline": false
    },
    {
      "model": "gpt-4o-mini",
      "cellKey": "production|gpt-4o-mini",
      "variantKind": "version",
      "variantLabel": "production",
      "promptVersionId": "4ccc4890-c7ca-4352-af8a-0a74896b1ff9",
      "isProductionBaseline": true
    }
  ],
  "exampleCount": 3,
  "results": {
    "total": 6,
    "succeeded": 6,
    "errored": 0
  }
}

Comparing against a run over a different dataset (status 400):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  "$ACRUXCORE_BASE_URL/runs/5ab85a08-19d3-4a26-9461-d98e2f4e8070/compare?base_cell=production%7Cmistral-small&target_run_id=80ddaa53-e87d-4a73-b914-551f08428896&target_cell=v2%7Cmistral-small"

{"error":{"code":"DATASET_MISMATCH","message":"The two runs graded different datasets, so their examples cannot be paired."}}

A cell key that is not in its run's grid returns 404 NOT_FOUND, and a missing base_cell or target_cell returns 400 VALIDATION_ERROR.

GET /api/v1/experiments

Lists the team's experiments. Note: unlike the JSDoc on ExperimentDto (which says runs is "omitted (empty array) on list"), the list endpoint as observed actually DOES populate runs per experiment (newest first) — this doc reflects the real observed response, not the comment.

curl
TypeScript
Python

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/experiments

const experiments = await hub.experiments.list();

experiments = await hub.experiments.list()

Response (status 200) — abbreviated to one experiment with its run. An experiment saved before judges were chosen per experiment shows no judgeModel in its config. Each run names the judge model it recorded at start and, when the experiment uses a custom judge prompt, the version it froze; both are null on a run created before judges were recorded:

{
  "data": [
    {
      "id": "8ee82248-e808-4730-8565-620204616307",
      "teamId": "8d3ceb9b-39d0-463b-8b91-aa2ef20ac9ba",
      "datasetId": "7792f4f8-01ad-443f-b5ed-322f107e6370",
      "promptId": "a290cf2f-a724-4389-8bea-da8b53229e00",
      "name": null,
      "config": {
        "models": ["gpt-4o-mini-or"],
        "judgeModel": "gpt-4o-mini-or",
        "versionIds": ["6059a265-eaff-4107-b98a-bf4808e630e5"],
        "judgePromptId": "7cbffcaf-d627-4b20-9945-4244badc1600"
      },
      "createdBy": "18c76b52-ee0d-4001-be8e-c29976488fbb",
      "createdAt": "2026-10-13T20:17:40.775Z",
      "runs": [
        {
          "id": "406968cd-9492-4967-b6f0-f7522d3fcbb9",
          "experimentId": "8ee82248-e808-4730-8565-620204616307",
          "status": "succeeded",
          "startedAt": "2026-10-13T20:17:40.802Z",
          "endedAt": "2026-10-13T20:17:47.193Z",
          "error": null,
          "createdAt": "2026-10-13T20:17:40.795Z",
          "judgeModel": "gpt-4o-mini-or",
          "judgePromptVersionId": "66ae6460-f8f8-4140-aa2b-5baef81c8fd3"
        }
      ]
    }
  ]
}

GET /api/v1/experiments/:id

Fetches one experiment with its runs (newest first).

curl
TypeScript
Python

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/experiments/9462c9b7-62a7-4f01-80fe-71f142cdd02b

const experiment = await hub.experiments.get('9462c9b7-62a7-4f01-80fe-71f142cdd02b');

experiment = await hub.experiments.get('9462c9b7-62a7-4f01-80fe-71f142cdd02b')

Response (status 200) — same shape as the list entry above for this experiment.

Error responses

Nonexistent experiment id (status 404):

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" $ACRUXCORE_BASE_URL/experiments/00000000-0000-0000-0000-000000000000

{ "error": { "code": "NOT_FOUND", "message": "Experiment not found." } }

DELETE /api/v1/experiments/:id

Deletes an experiment and every run under it, including each run's cells and reports. The dataset it swept and any prompt version promoted out of one of its runs are separate resources and are untouched; so is any optimizer candidate the runs drafted (the candidate's run link is simply cleared).

Refused while any of the experiment's runs is still queued or running — the worker is mid-flight and would keep writing results for a row that no longer exists. Wait for the run to settle, or delete the settled runs individually with DELETE /api/v1/runs/:id.

curl
TypeScript
Python

curl -X DELETE -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  $ACRUXCORE_BASE_URL/experiments/ff661325-588f-4984-8825-6102acfe3161

await hub.experiments.delete('ff661325-588f-4984-8825-6102acfe3161');

await hub.experiments.delete('ff661325-588f-4984-8825-6102acfe3161')

Response (status 200):

{ "success": true }

Error responses

A run is still in flight (status 409):

curl -X DELETE -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  $ACRUXCORE_BASE_URL/experiments/608693aa-e344-4c34-a5c0-12d9d18cbbb5

{
  "error": {
    "code": "RUN_IN_FLIGHT",
    "message": "This experiment has 1 run still queued or running. Wait for it to finish, then delete."
  }
}

Already deleted, nonexistent, or another team's id (status 404 — never confirms whether it exists elsewhere):

curl -X DELETE -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  $ACRUXCORE_BASE_URL/experiments/ff661325-588f-4984-8825-6102acfe3161

{ "error": { "code": "NOT_FOUND", "message": "Experiment not found." } }

POST /api/v1/runs/:id/rejudge

Sends a finished run's existing outputs back to its judge. The models under test are not called again, so only the judge calls are billed. Use it when the judge provider failed or ran out of quota part-way through a run, and the run ended with results that have no score.

The judge grades with what the run started with: the judge model, the custom judge prompt version if there was one, each example's criteria and history, and the dataset's overall feedback. Editing the dataset after the run does not change what a re-judge grades against.

The run goes back to running. The chosen results lose their old verdicts, and the run settles again as succeeded or failed when the new verdicts are in. Poll GET /runs/:id to follow it.

curl
TypeScript
Python

curl -X POST $ACRUXCORE_BASE_URL/runs/c4c624f8-5087-4ca3-8903-bc0ed08c126f/rejudge \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"only_unscored": false}'

await hub.runs.rejudge('c4c624f8-5087-4ca3-8903-bc0ed08c126f', { onlyUnscored: false });

await hub.runs.rejudge('c4c624f8-5087-4ca3-8903-bc0ed08c126f', only_unscored=False)

Response (status 202):

{"run_id":"c4c624f8-5087-4ca3-8903-bc0ed08c126f","status":"running","rejudging":3}

With an empty body, only results that have no score are sent. When every result already has a score, nothing is queued and the run is left as it was:

curl -X POST $ACRUXCORE_BASE_URL/runs/c4c624f8-5087-4ca3-8903-bc0ed08c126f/rejudge \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

Response (status 200):

{"run_id":"c4c624f8-5087-4ca3-8903-bc0ed08c126f","status":"succeeded","rejudging":0}

Fields

Field	Required	Notes
`only_unscored`	no	Defaults to `true`: grade only results with no score. `false` grades every result that has an output again, and replaces the existing scores. A result whose model call failed has no output and is never sent.

Response

Field	Notes
`run_id`	The run.
`status`	`running` when judge calls were queued; the run's unchanged status when none were.
`rejudging`	How many results were sent to the judge.

Error responses

The run has not finished (status 409):

{"error":{"code":"RUN_IN_FLIGHT","message":"This run is still running. Wait for it to finish, then re-judge it."}}

only_unscored is not a boolean (status 400):

{"error":{"code":"VALIDATION_ERROR","message":"Expected boolean, received string"}}

Some cells of the run never produced a result, for example because the run failed before they ran (status 409). A re-judge grades existing results only, so start a new run instead:

{"error":{"code":"RUN_INCOMPLETE","message":"This run has 2 of 3 results, because some cells never ran. A re-judge grades existing results only. Start a new run instead."}}

A judge call for one of the run's results is still queued or waiting to retry (status 409):

{"error":{"code":"JUDGE_PENDING","message":"A judge job for this run is still queued or retrying. Wait for it to settle, then re-judge."}}

The run was made by runLocal / run_local, so its scores come from your own code (status 409):

{"error":{"code":"LOCAL_EXPERIMENT","message":"This is a local run: its scores come from your own code through the SDK, so there is no server judge to run again."}}

The run's judge model has been removed from the team since the run (status 400):

{"error":{"code":"VALIDATION_ERROR","message":"Model 'judge-639' is not registered. Add it under Gateway → Models."}}

DELETE /api/v1/runs/:id

Deletes one run and every cell it produced, so its report goes with it. The parent experiment stays — a run is one execution of it — and so does any optimizer candidate the run drafted, which matters because a candidate may already have been promoted to a real prompt version.

Refused while the run is still queued or running, for the same reason as the experiment delete above.

curl
TypeScript
Python

curl -X DELETE -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  $ACRUXCORE_BASE_URL/runs/f595cade-4145-49a9-b228-aa344dad15c8

await hub.runs.delete('f595cade-4145-49a9-b228-aa344dad15c8');

await hub.runs.delete('f595cade-4145-49a9-b228-aa344dad15c8')

Response (status 200):

{ "success": true }

A subsequent read of the same run 404s, while its experiment still resolves:

curl -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  $ACRUXCORE_BASE_URL/runs/f595cade-4145-49a9-b228-aa344dad15c8

{ "error": { "code": "NOT_FOUND", "message": "Run not found." } }

Error responses

The run has not finished (status 409):

{
  "error": {
    "code": "RUN_IN_FLIGHT",
    "message": "This run is still queued. Wait for it to finish, then delete it."
  }
}

POST /api/v1/experiments​

Error responses​

Custom judge prompt (judge_prompt_id field)​

Alias-based baseline (alias field)​

Prompt-mismatch warning (promptMismatchWarning, response only)​

POST /api/v1/experiments/:id/runs​

Error responses​

GET /api/v1/runs​

Error responses​

GET /api/v1/runs/:id​

Error responses​

GET /api/v1/runs/:id/report​

Which judge graded the run​

Cells with failed or unscored examples​

Error responses​

Which version is behind each variant​

GET /api/v1/runs/:id/cells/:cellKey​

A cell that failed​

Error responses​

GET /api/v1/runs/:id/compare​

GET /api/v1/experiments​

GET /api/v1/experiments/:id​

Error responses​

DELETE /api/v1/experiments/:id​

Error responses​

POST /api/v1/runs/:id/rejudge​

Error responses​

DELETE /api/v1/runs/:id​

Error responses​

POST /api/v1/experiments

Error responses

Custom judge prompt (`judge_prompt_id` field)

Alias-based baseline (`alias` field)

Prompt-mismatch warning (`promptMismatchWarning`, response only)

POST /api/v1/experiments/:id/runs

Error responses

GET /api/v1/runs

Error responses

GET /api/v1/runs/:id

Error responses

GET /api/v1/runs/:id/report

Which judge graded the run

Cells with failed or unscored examples

Error responses

Which version is behind each variant

GET /api/v1/runs/:id/cells/:cellKey

A cell that failed

Error responses

GET /api/v1/runs/:id/compare

GET /api/v1/experiments

GET /api/v1/experiments/:id

Error responses

DELETE /api/v1/experiments/:id

Error responses

POST /api/v1/runs/:id/rejudge

Error responses

DELETE /api/v1/runs/:id

Error responses