Skip to main content

Evaluate a prompt against a dataset

What you'll build: a dataset made from real feedback or from a test set you already keep, an experiment that runs one or more (version × model) combinations across it, and a report comparing them — so "did this prompt change actually help?" has a data-backed answer.

Requires the worker

Experiment runs are processed asynchronously by the AcruxCore worker process. The API calls below return immediately with a queued status; the worker performs the actual gateway calls and scoring in the background.

1. Collect feedback​

If you already have a test set in a CSV or JSON file, skip to step 2, from a file. Otherwise, start from signal on real traffic. On any trace (or span), leave a thumbs up/down and an optional comment — that comment becomes the example's grading criteria later. Feedback is only eligible for a dataset if its source trace captured payloads (so the prompt variables were stored). If your app renders the prompt itself and sends prompt_version_id with its own messages, send the values it rendered with as variables as well — those variables are what an example replays.

Feedback page

Your app can report feedback too, so a thumbs-down in your own UI lands here without anyone opening the dashboard. The trace id comes back from the call you are rating: the SDK returns it as result.gateway.traceId (result.trace_id from a tool loop), and a raw HTTP call reads it from the x-gateway-trace-id response header — see chat, stream, and collect feedback with the SDK for the full round trip. source records who the rating came from — end_user for your app's users, developer for your own team; it defaults to user.

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

const feedback = await hub.traces.submitFeedback({
  traceId: '<trace-id>',
  rating: -1,
  comment: 'Too formal — be warmer',
  source: 'end_user',
});
console.log(feedback.id);

2. Build a dataset​

A dataset is a frozen set of example inputs, each with its own criteria: the prompt variables to render, plus what a judge should check in the reply. Evaluations → Datasets lists them, and a new workspace starts empty.

Empty Evaluations / Datasets page

A dataset can be filled in three ways. Steps 3 to 5 work the same for all three.

  • From feedback — select the feedback rows from step 1.
  • From a file — import a test set you already keep in a CSV or JSON file.
  • By hand — type each row in the dashboard.

From feedback​

Select feedback rows and turn them into a dataset. Each row brings the variables the call was made with, and its comment becomes the example's criteria.

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

const dataset = await hub.datasets.buildFromFeedback({
  name: 'unhappy-support-replies',
  overallFeedback: 'Warm, concise, always end with a next step',
  feedbackIds: ['<feedback-id-1>', '<feedback-id-2>'],
});
console.log(dataset.id, dataset.exampleCount);

From a file​

In the dashboard, New dataset creates an empty dataset. On that dataset, open Add example and choose the Import file tab. In a CSV, each column becomes a prompt variable, and a column named criteria holds the judge criteria:

question,customer_tier,criteria
How long do I have to return an item?,free,"Says 30 days from delivery, and that items must be unused."
Can I get a refund on an annual plan?,pro,Quotes the 14-day refund window for annual plans.

A JSON file is an array of objects in the same shape, and it may also carry a history of earlier chat turns. The tab shows a preview and lists any row with a problem by its row number. It imports up to 500 rows at a time.

From code, create the dataset and send the rows in one call. The call is all-or-nothing: if one row is invalid, nothing is written and the error names the row.

const dataset = await hub.datasets.create({ name: 'support-test-set' });

const result = await hub.datasets.addExamples(dataset.id, [
  { input: { question: 'How long do I have to return an item?' }, criteria: 'Says 30 days from delivery, and that items must be unused.' },
  { input: { question: 'Can I get a refund on an annual plan?' }, criteria: 'Quotes the 14-day refund window for annual plans.' },
]);
console.log(result); // { added: 2, exampleCount: 2 }

The full request and error shapes are in the API reference.

By hand​

New dataset creates an empty dataset, and Add example on it writes one row: the prompt variables, plus the criteria a judge should check. Use this path for a prompt that has no traffic yet and no test file.

3. Create and run an experiment​

An experiment pairs a dataset with a prompt under test and an explicit grid to sweep: one or more version_ids × one or more models. Creating it is one call; starting a run is another (it returns 202 queued and hands off to the worker).

Every experiment also names a judge model: the model that grades each reply against the example criteria. AcruxCore has no default judge, so you pick one of the models registered under Gateway → Models. It can be a model you are testing, as below, or a stronger one. In the dashboard it is the required Judge model field on the run page.

const experiment = await hub.experiments.create({
  datasetId: '<dataset-id>',
  promptId: '<prompt-id>',
  name: 'support-reply v2 vs v1',
  versionIds: ['<v2-version-id>'],
  models: ['support-model'],
  judgeModel: 'support-model',
});

const run = await hub.experiments.startRun(experiment.id);
console.log(run.runId, run.status); // -> queued

When a prompt is under test, AcruxCore also adds an automatic baseline cell, so you always compare your candidate against something rather than nothing — by default that's the prompt's production alias, falling back to its latest committed version if there's no production alias yet. Pick a different alias in the dashboard's Baseline alias field (or pass "alias": "staging" over the API) if you want the comparison to be against staging, dev, or any other alias instead.

If the dataset's examples were captured against a different prompt than the one under test here, the run still starts — AcruxCore just adds a warning to the response and the run page so you know the comparison might not be apples-to-apples. It never blocks the run.

Grade with your own judge prompt​

The built-in judge reads the criteria as a correction to an earlier, wrong reply. If your criteria are plain expected values, or you want the judge to follow your own rules, write a judge prompt and pass its id as judge_prompt_id. A judge prompt is an ordinary prompt with a committed version. Its templates can use four variables:

  • {{ output }} is the reply being graded.
  • {{ criteria }} is the example's criteria.
  • {{ overallFeedback }} is the dataset's overall feedback.
  • {{ history }} is the earlier conversation as JSON.

A missing value is an empty string. A custom judge grades every example, even one with no criteria, so keep a rubric in the prompt itself.

For example, a version with this system message:

You grade support replies. The criteria list the facts a correct reply contains.
Ignore tone and formatting. Score 100 only if every fact is present.
Criteria: {{ criteria }}

and this user message: Reply to grade: {{ output }}.

AcruxCore always adds its own output-format message after your template, so your prompt does not need to describe the JSON score format. For a full example with a rubric, see Write a custom judge prompt.

const experiment = await hub.experiments.create({
  datasetId: '<dataset-id>',
  promptId: '<prompt-id>',
  versionIds: ['<v2-version-id>'],
  models: ['support-model'],
  judgeModel: 'support-model',
  judgePromptId: '<judge-prompt-id>',
});

In the dashboard, pick the prompt in the Judge prompt field under Judge model. Leave it empty to use the built-in judge.

Each run grades with the judge prompt's production version, or its latest version if it has no production alias. The version is fixed when the run starts, so promoting a new judge version changes the next run, not the one in progress.

4. Read the report​

Poll the run until it finishes, then read the report — per-cell scores across the dataset, so you can see whether v2 beat v1 (or the baseline).

const detail = await hub.runs.get('<run-id>'); // status: queued → running → completed

const report = await hub.runs.getReport('<run-id>'); // per-cell scores

In the web app, Evaluations → Runs lists every run your team has made — newest first, with each run's score, best variant and duration — and each row opens the same report as a comparison grid. The same history is available over the API:

const { data: recentRuns, total } = await hub.runs.list({ limit: 20 });

const datasetRuns = await hub.runs.list({ datasetId: '<dataset-id>' });

See which examples changed​

A cell's average can hide what happened to single examples. A new version can fix three examples and break two, and its average still goes up. The compare view puts two cells side by side, one row per example, and counts the examples that went from fail to pass (Improved) and from pass to fail (Regressed).

On a run report, click Compare per example. The page opens on the baseline against the best other cell, and shows only the examples that changed. Click a counter to show another group, and click a row to read both outputs and both judge reasons. To compare with an earlier run over the same dataset, pick it under Target.

The Compare per example page. The base is production · mistral-small and the target is v2 · mistral-small. The counters read Changed 5, Improved 3, Regressed 2, Unchanged 25. The table lists two regressed examples, with pass at 90 and 80 becoming fail at 60, and three improved examples.

5. Use this as a gate before you promote​

The automatic baseline cell from step 3 is what makes this a regression gate, not just a comparison: every run already scores your candidate version and whatever the baseline alias currently serves (production by default), side by side, on the same dataset. Read the report before promoting a version, not after:

  • Candidate clearly beats baseline → promote it, per Version a prompt.
  • Candidate ties or loses on one or more examples → that's a regression the dataset caught before a real user did. Compare per example lists those examples, even when the average went up. Don't promote; either fix the version or revisit the dataset's criteria if the loss looks like a scoring artifact rather than a real quality drop.

Because the dataset is frozen (built once in step 2), the comparison is stable — re-running the same experiment against a later candidate version still checks it against the same examples and the same criteria, so results are comparable across runs over time.

What's next​