Skip to main content

Evaluate a prompt with conversation history

What you'll build: a dataset example that carries the earlier turns of a real conversation alongside the flagged reply, and a run that replays those turns ahead of the new one — so the judge and the optimizer see the same context the model saw, not just the single message someone reacted to.

Evaluate a prompt against a dataset and Improve a prompt from feedback both start from a single flagged reply. That is enough when a prompt answers one question at a time. It stops being enough once calls are grouped into a session — a multi-turn conversation — because a reply can be wrong only in light of what was said earlier. Score that reply on its own and a judge (or a person) has no way to tell "this failed" from "this failed because it forgot the last turn." AcruxCore reconstructs the prior turns of the flagged trace's session automatically, so the dataset example carries the conversation, not just its last line.

1. Leave feedback on a reply from a multi-turn session​

Reconstruction needs two things from the flagged trace: a session_id — the same one every call in the conversation was tagged with, either the gateway's x-session-id header or an SDK call's trace: { sessionId } option — and captured payloads, same as any dataset-eligible feedback (see Trace an LLM call for the capture setting). Select the row on Observability → Feedback as usual.

The Feedback page with two thumbs-down rows selected from a multi-turn session, and the Create dataset action visible

2. Build a dataset — the history comes along automatically​

Click Create dataset. Nothing here differs from a single-turn dataset: name it, optionally add an overall rubric, and confirm.

The Create dataset from feedback dialog with a name and overall-feedback rubric filled in

The difference shows up on the dataset itself. Each example whose source trace had a session gets a History column — collapsed by default, so a single-turn dataset looks exactly like it always has:

The dataset detail page with the History column's "2 prior messages" disclosure expanded, showing the reconstructed user/assistant turns

That disclosure is reading DatasetExample.history — a plain array of { role, content } messages, oldest first, walked backward from the flagged trace through every earlier trace sharing its session_id. It degrades to nothing instead of failing the build: a first-turn trace, a trace with no session, or one whose earlier spans hold nothing readable to replay all just leave history empty, and the example is built the same as before this existed.

3. Run it — history replays ahead of the new turn​

Starting a run works exactly as in Evaluate a prompt against a dataset: pick the prompt under test, a version, and a model.

The run report showing one variant scored against the production baseline

What's different is invisible until you open a cell. At run-start, each example's history is frozen alongside its input and criteria. When the cell runs, that frozen history is spliced in right after the version's system message and right before the new turn's rendered input — so the model answers with the same conversation in front of it that produced the original reply, not the bare new question in isolation. The same frozen history is also handed to the judge as grading context, so "does this refer back correctly?" is something the judge can actually check instead of guessing from the criteria text alone.

The cell drill-down panel with a "2 prior messages" disclosure expanded, showing input, output, and judge reasoning that references the earlier turn

Read the judge's reasoning here the same way Improve a prompt from feedback recommends: it is grading a specific claim ("does the output correctly resolve the back-reference to the prior turn"), not a vibe. A partial-credit score — the model in the screenshot named the wrong scope (a city instead of the country it belongs to) while still clearly using the prior turn — is exactly the kind of nuance a single-turn eval could never have caught, because there wouldn't have been a prior turn to get right or wrong.

4. Add history by hand, when there's no live trace to reconstruct from​

Reconstruction only runs for feedback built from a real session. Building an example from a transcript you already have — a support log, a recorded incident, a hand-written test case — has no source trace at all, so add history directly on a manually-added example instead:

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

await hub.datasets.addExample('<dataset-id>', {
  input: { question: 'What language do people speak there?' },
  criteria: 'Must name Italian and refer to Rome/Italy from the prior turn, not ask which place.',
  history: [
    { role: 'user', content: 'What is the capital of Italy?' },
    { role: 'assistant', content: 'The capital of Italy is Rome.' },
  ],
});

history is capped at 20 messages / 32 KB — plenty for a real back-and-forth, tight enough that one runaway example can't blow up every run built from it.

Tool calls replay too, minus a dangling one

A reconstructed or hand-supplied history can include tool_calls and tool-role results, not just plain text turns — a prior turn that called a tool replays with the call and its result intact. The one thing that gets trimmed automatically is a trailing unanswered tool_calls message (a prior turn abandoned mid tool-loop): replaying that verbatim is a guaranteed 400 from most providers, so the splice drops it rather than failing the cell.

Doing this over the API​

The dashboard covers building a dataset from feedback and reading a run's history back. Everything else — a manual example with explicit history, and the experiment/run calls themselves — works through the SDK too:

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

const dataset = await hub.datasets.buildFromFeedback({
  name: 'geography-followups',
  overallFeedback:
    'When a question refers back to something asked earlier in the conversation, ' +
    'answer about that earlier subject, not a fresh unrelated one.',
  feedbackIds: ['<feedback-id>'],
});

const full = await hub.datasets.get(dataset.id);
console.log(full.examples[0].history);

const experiment = await hub.experiments.create({
  datasetId: dataset.id,
  promptId: '<prompt-id>',
  name: 'history replay check',
  versionIds: ['<version-id>'],
  models: ['gpt-4o-mini'],
});

const run = await hub.experiments.startRun(experiment.id);
console.log(run.runId, run.status); // -> queued

const cell = await hub.runs.getCell(run.runId, 'v1|gpt-4o-mini');
console.log(cell.examples[0].history);

What's next​