Local experiments API

A server-side experiment runs one stored prompt per example. A local experiment runs your own code instead: the SDK calls your app on each dataset example, calls your scorers on each output, and uploads the results here. The result is an ordinary run, so the run list, the report and Compare per example show it.

You will usually not call these endpoints yourself. experiments.run_local (Python) and experiments.runLocal (Node) make all three calls for you. See the guide Run your app as an experiment.

The flow is three calls:

POST /experiments/local creates the experiment and a running run, and returns the frozen examples.
POST /runs/:id/results uploads results, up to 100 per call, as many times as needed.
POST /runs/:id/finish closes the run.

All three need owner, admin or editor. A team-scoped API key holds no role and is refused; use a personal key. Scores on the wire are 0–100, like the LLM judge. The SDKs take 0–1 from your scorers and convert.

A run that stops uploading — your script crashed before finish — is marked failed by the stale-run sweep after 120 minutes without a new result. The SDKs upload finished results at least once a minute, so a run of slow examples is not mistaken for a crashed one.

POST /api/v1/experiments/local

Body:

dataset_id (required) — the dataset to run on. It must have at least one example.
name (required) — labels the run, for example v1. Becomes the experiment name and the cell's variant label. At most 100 characters, no |.
model — a free-text label for the cell, default local. It is not checked against Gateway → Models, because your app may call several models.
primary_score — the score the report, the run list and Compare per example use by default. Omitted, each result's first score is used, which is the first scorer's when scores keep scorer order. Named, every result that has scores must include it (see the errors below).
metadata — any JSON object, stored on the experiment (a git commit, a config name).

curl -X POST $ACRUXCORE_BASE_URL/experiments/local \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dataset_id":"bdefa344-5281-433c-97fe-aee0f06fdc90","name":"v1","model":"gpt-4o-mini-or","primary_score":"team_exact"}'

Response (status 201):

{
  "experiment_id": "1c0486a0-50df-466f-99f6-3a7f4d881a8a",
  "run_id": "14f59c48-e618-41e6-a0a3-8b415c3c065c",
  "cell_key": "v1|gpt-4o-mini-or",
  "url": "http://localhost:5596/evaluations/runs/14f59c48-e618-41e6-a0a3-8b415c3c065c",
  "examples": [
    {
      "id": "075e5ccd-8a97-4d8c-9ee4-be65920b5333",
      "input": { "ticket": "I was charged twice for my October invoice." },
      "criteria": null,
      "expected": { "team": "finance", "category": "billing" },
      "history": null
    },
    {
      "id": "29a3c6d0-87a7-49ef-ba37-a2d38826b971",
      "input": { "ticket": "And the second one?" },
      "criteria": null,
      "expected": { "team": "finance", "category": "billing" },
      "history": [
        { "role": "user", "content": "Which invoices are unpaid?" },
        { "role": "assistant", "content": "October and November." }
      ]
    }
  ]
}

url is the run's page in the dashboard. It is null when the server has no APP_URL. history holds the earlier turns of an example built from a multi-turn session, or null. A server-side run replays it before the input; your app can do the same.

Error responses

Status	Code	When
400	`VALIDATION_ERROR`	`name` missing or contains `\|`, or the dataset has no examples
404	`NOT_FOUND`	The dataset is not in your team

POST /api/v1/runs/:id/results

Uploads 1 to 100 results. Each result:

example_id (required) — an example id from the create response.
output — what your app returned, any JSON up to 64 KB.
trace_id — the trace that holds this example's spans, so the dashboard can link it.
error — set when your app raised; leave scores empty then. The result's output is stored as empty.
scores — [{ name, score, passed?, reason? }]. score is 0–100, or null when the scorer itself failed, and is stored rounded to a whole number, like a judge score. passed defaults to score >= 50. Names must be unique per result.

Sending a result for an example that already has one replaces it, so a retried upload never creates a duplicate. That holds when the retry arrives while the first try is still being written: uploads to one run are written one at a time.

curl -X POST $ACRUXCORE_BASE_URL/runs/14f59c48-e618-41e6-a0a3-8b415c3c065c/results \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"results":[
    {"example_id":"075e5ccd-8a97-4d8c-9ee4-be65920b5333","output":{"category":"billing","team":"finance"},
     "scores":[{"name":"category_exact","score":100},{"name":"team_exact","score":100}]},
    {"example_id":"29a3c6d0-87a7-49ef-ba37-a2d38826b971","output":{"category":"question","team":"support"},
     "scores":[{"name":"category_exact","score":0,"reason":"expected billing, got question"},
               {"name":"team_exact","score":0,"reason":"expected finance, got support"}]}]}'

Response (status 200):

{ "written": 2, "completed": 2, "total": 2 }

completed is how many of the run's examples have a result so far; total is how many the run froze.

Error responses

Status	Code	When
400	`VALIDATION_ERROR`	An `example_id` is not in this run or appears twice in the batch; a score is outside 0–100; a scored result lacks the run's `primary_score`
404	`NOT_FOUND`	The run is not in your team
409	`RUN_NOT_LOCAL`	The run was started on the server; its results come from the run engine
409	`RUN_NOT_RUNNING`	The run has already finished

{ "error": { "code": "RUN_NOT_RUNNING", "message": "This run has already finished, so it no longer accepts results. Start a new run." } }

A result without the named primary score, on the run created above with "primary_score":"team_exact":

curl -X POST $ACRUXCORE_BASE_URL/runs/14f59c48-e618-41e6-a0a3-8b415c3c065c/results \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"results":[{"example_id":"075e5ccd-8a97-4d8c-9ee4-be65920b5333","output":{"category":"billing"},
       "scores":[{"name":"category_exact","score":100}]}]}'

Response (status 400):

{ "error": { "code": "VALIDATION_ERROR", "message": "results[0] has no score named \"team_exact\", the run's primary score. Its scores are: category_exact." } }

POST /api/v1/runs/:id/finish

Closes the run and returns its detail. The run ends failed when the body carries an error (the SDK sends one when its own loop broke), when no result was uploaded, or when every example errored. Otherwise it ends succeeded. No email is sent.

curl -X POST $ACRUXCORE_BASE_URL/runs/14f59c48-e618-41e6-a0a3-8b415c3c065c/finish \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'

Response (status 200):

{
  "id": "14f59c48-e618-41e6-a0a3-8b415c3c065c",
  "kind": "local",
  "experimentId": "1c0486a0-50df-466f-99f6-3a7f4d881a8a",
  "datasetId": "bdefa344-5281-433c-97fe-aee0f06fdc90",
  "status": "succeeded",
  "startedAt": "2026-10-16T14:28:08.246Z",
  "endedAt": "2026-10-16T14:28:41.359Z",
  "error": null,
  "createdAt": "2026-10-16T14:28:08.247Z",
  "grid": [
    { "model": "gpt-4o-mini-or", "cellKey": "v1|gpt-4o-mini-or", "variantKind": "local", "variantLabel": "v1", "isProductionBaseline": false }
  ],
  "exampleCount": 2,
  "results": { "total": 2, "succeeded": 2, "errored": 0 }
}

A second finish, or a finish after the stale-run sweep failed the run, answers 409 RUN_NOT_RUNNING.

POST /api/v1/runs/:id/judge

Grades one output of a local run with an LLM judge, on the server. The judge reads the example's criteria and the dataset's overall feedback as they were when the run started, the same inputs a server-side run's judge reads. Nothing is stored on the run: the SDK puts the returned score in the result it uploads. experiments.judge(model) in both SDKs is a scorer that calls this endpoint. Editor role or above required.

Body: example_id (an example of this run), output (any JSON value), model (a model name from your Models list), and optionally judge_prompt_id to grade with one of your judge prompts. A run grades every output with one version of that judge prompt: by default, the version its production alias serves at the run's first judge call. judge_prompt_version names the version instead. Pass the version an optimize run graded with, and every run you compare against it grades the same way.

curl -X POST $ACRUXCORE_BASE_URL/runs/9f351ec4-efb4-4456-9be1-e9cb2c838675/judge \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"example_id":"0253bd73-1aa8-4bd4-81cc-23da83f95a4e","output":"Yes. BR-1077 was delivered on 20 October, so the 14-day window for a used grinder is open. A used grinder is refunded as store credit, not money. Reply to your order confirmation email to start the return.","model":"gpt-4.1"}'

Response (status 200) — score is 0–100, and judge_trace_id is the judge's own trace:

{
  "score": 95,
  "passed": true,
  "reason": "The output correctly states that the 14-day window for returning a used grinder is open, specifies that the refund will be as store credit, and instructs the customer to reply to the order confirmation email to start the return. The only minor omission is not explicitly stating that the delivery was 6 days ago, but this does not affect the correctness or clarity of the answer.",
  "judge_trace_id": "f02ae94d-c5ca-478d-bce5-138f3591e112"
}

Grading with version 1 of a judge prompt, whatever production serves now:

curl -X POST $ACRUXCORE_BASE_URL/runs/93aef7df-009c-4159-8240-7229dbc2d897/judge \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"example_id":"79e5e814-2a72-446a-b408-9d492dc21dda","output":"Yes. The kettle from BR-1088 is unopened, so you can return it within 30 days of delivery for a full refund. Reply to your order confirmation email to start the return.","model":"gpt-4.1","judge_prompt_id":"29d3bb49-a6f4-4592-a4b1-c59d28c3aab5","judge_prompt_version":1}'

Response (status 200):

{
  "score": 100,
  "passed": true,
  "reason": "The answer correctly confirms eligibility for a full refund within the 30-day window, specifies the item is unopened, and instructs the customer to reply to the order confirmation email to start the return, matching all requirements.",
  "judge_trace_id": "91a26e5b-421c-4ce3-bb85-5f88ea2725be"
}

409 — the run already grades with another version of that judge prompt. Its scores would mix two rubrics, so start a new run instead:

{"error":{"code":"JUDGE_PROMPT_VERSION_MISMATCH","message":"This run already grades with another version of this judge prompt. Start a new run to grade with version 2."}}

400 — the judge prompt has no such version:

{"error":{"code":"VALIDATION_ERROR","message":"Judge prompt 'brewline-judge-e2e' has no version 9."}}

400 — judge_prompt_version without judge_prompt_id:

{"error":{"code":"VALIDATION_ERROR","message":"judge_prompt_version needs judge_prompt_id: it names a version of that judge prompt."}}

400 — the run id in the path is not a UUID:

{"error":{"code":"VALIDATION_ERROR","message":"id must be a valid UUID."}}

409 — the run was started on the server, which grades with its own judge:

{"error":{"code":"RUN_NOT_LOCAL","message":"This run was started on the server, and its own judge grades it. Only a local run takes judge calls from the SDK."}}

400 — the example is not part of this run:

{"error":{"code":"VALIDATION_ERROR","message":"example_id 00000000-0000-0000-0000-000000000000 is not an example of this run."}}

400 — the model is not in your Models list:

{"error":{"code":"VALIDATION_ERROR","message":"Model 'no-such-model' is not registered. Add it under Gateway → Models."}}

Reading a local run

The read endpoints in Experiments & Runs work on a local run as on any other. What a local run adds:

GET /runs and GET /runs/:id report kind: "local".
GET /runs/:id/report cells carry namedScores, one aggregate per scorer. avgScore and passRate on the cell are the primary score's.

"namedScores": [
  { "name": "category_exact", "avgScore": 50, "passRate": 0.5, "scoredCount": 2 },
  { "name": "team_exact", "avgScore": 50, "passRate": 0.5, "scoredCount": 2 }
]

GET /runs/:id/cells/:cellKey examples carry scores (every named score with its reason) and expected.
GET /runs/:id/compare takes score=<name> to classify on a named score instead of the primary one, and returns scoreNames for a picker. A name that neither cell has answers 404 with its own code:

curl -G $ACRUXCORE_BASE_URL/runs/14f59c48-e618-41e6-a0a3-8b415c3c065c/compare \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  --data-urlencode "base_cell=v1|gpt-4o-mini-or" \
  --data-urlencode "target_cell=v1|gpt-4o-mini-or" \
  --data-urlencode "score=nope"

Response (status 404):

{ "error": { "code": "SCORE_NOT_FOUND", "message": "Neither cell has a score named \"nope\"." } }

POST /experiments/:id/runs on a local experiment answers 409 LOCAL_EXPERIMENT: the server has no code to run for it.

POST /api/v1/experiments/local​

Error responses​

POST /api/v1/runs/:id/results​

Error responses​

POST /api/v1/runs/:id/finish​

POST /api/v1/runs/:id/judge​

Reading a local run​

POST /api/v1/experiments/local

Error responses

POST /api/v1/runs/:id/results

Error responses

POST /api/v1/runs/:id/finish

POST /api/v1/runs/:id/judge

Reading a local run