Skip to main content

Evaluate a prompt against a dataset

What you'll build: a dataset made from real feedback, an experiment that runs one or more (version × model) combinations across it, and a report comparing them — so "did this prompt change actually help?" has a data-backed answer.

Requires the worker

Experiment runs are processed asynchronously by the AcruxCore worker process. The API calls below return immediately with a queued status; the worker performs the actual gateway calls and scoring in the background.

1. Collect feedback​

Evaluation starts from signal on real traffic. On any trace (or span), leave a thumbs up/down and an optional comment — that comment becomes the example's grading criteria later. Feedback is only eligible for a dataset if its source trace captured payloads (so the prompt variables were stored). If your app renders the prompt itself and sends prompt_version_id with its own messages, send the values it rendered with as variables as well — those variables are what an example replays.

Feedback page

Feedback can also be reported over the API:

curl -X POST "$ACRUXCORE_BASE_URL/traces/<traceId>/feedback" \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rating": -1, "comment": "Too formal — be warmer"}'

2. Build a dataset from feedback​

Select feedback rows and turn them into a dataset — a frozen set of example inputs, each with its own criteria. Evaluations → Datasets lists them; a new workspace starts empty.

Empty Evaluations / Datasets page

No feedback yet?

A brand-new workspace has no traffic to collect feedback on, so start from the other end: New dataset on that page creates an empty one, and Add example on the dataset writes a row by hand — the prompt variables to render, plus the criteria a judge should check. The rest of this guide works the same either way.

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

const dataset = await hub.datasets.buildFromFeedback({
  name: 'unhappy-support-replies',
  overallFeedback: 'Warm, concise, always end with a next step',
  feedbackIds: ['<feedback-id-1>', '<feedback-id-2>'],
});
console.log(dataset.id, dataset.exampleCount);

3. Create and run an experiment​

An experiment pairs a dataset with a prompt under test and an explicit grid to sweep: one or more version_ids × one or more models. Creating it is one call; starting a run is another (it returns 202 queued and hands off to the worker).

const experiment = await hub.experiments.create({
  datasetId: '<dataset-id>',
  promptId: '<prompt-id>',
  name: 'support-reply v2 vs v1',
  versionIds: ['<v2-version-id>'],
  models: ['support-model'],
});

const run = await hub.experiments.startRun(experiment.id);
console.log(run.runId, run.status); // -> queued

When a prompt is under test, AcruxCore also adds an automatic baseline cell, so you always compare your candidate against something rather than nothing — by default that's the prompt's production alias, falling back to its latest committed version if there's no production alias yet. Pick a different alias in the dashboard's Baseline alias field (or pass "alias": "staging" over the API) if you want the comparison to be against staging, dev, or any other alias instead.

If the dataset's examples were captured against a different prompt than the one under test here, the run still starts — AcruxCore just adds a warning to the response and the run page so you know the comparison might not be apples-to-apples. It never blocks the run.

4. Read the report​

Poll the run until it finishes, then read the report — per-cell scores across the dataset, so you can see whether v2 beat v1 (or the baseline).

const detail = await hub.runs.get('<run-id>'); // status: queued → running → completed

const report = await hub.runs.getReport('<run-id>'); // per-cell scores

In the web app, Evaluations → Runs lists every run your team has made — newest first, with each run's score, best variant and duration — and each row opens the same report as a comparison grid. The same history is available over the API:

const { data: recentRuns, total } = await hub.runs.list({ limit: 20 });

const datasetRuns = await hub.runs.list({ datasetId: '<dataset-id>' });

5. Use this as a gate before you promote​

The automatic baseline cell from step 3 is what makes this a regression gate, not just a comparison: every run already scores your candidate version and whatever the baseline alias currently serves (production by default), side by side, on the same dataset. Read the report before promoting a version, not after:

  • Candidate clearly beats baseline → promote it, per Version a prompt.
  • Candidate ties or loses on one or more examples → that's a regression the dataset caught before a real user did. Don't promote; either fix the version or revisit the dataset's criteria if the loss looks like a scoring artifact rather than a real quality drop.

Because the dataset is frozen (built once from real feedback in step 2), the comparison is stable — re-running the same experiment against a later candidate version still checks it against the same examples and the same criteria, so results are comparable across runs over time.

What's next​