Experiments

An experiment runs a task across every row of a dataset and scores each result, producing a reproducible record you can compare against later runs. It's offline evaluation: the loop where you catch a regression before your users do.

Open Experiments in the sidebar, or reach it from a dataset or prompt via Evaluate → Run in Experiment.

The task is a run-time choice

The dataset and the task are chosen independently. A dataset carries named inputs and ground truth; what you run against it is decided per experiment. There are two places a task can come from:

• A prompt: pick a prompt and a version. The experiment compiles that exact version with each row's variables and sends it to your model(s).
• Raw dataset input: no prompt. The row already carries its whole input, so it goes to the model as-is. Use this when the text you want to test is already in the dataset. If the row carries messages, the conversation is sent as-is: the last user turn is the question, earlier turns are context.

The resolved prompt is pinned by its immutable version_hash, so an experiment always records the exact version that ran, even if you later move the production label to a different version. Reproducibility is the whole point.

A third source, external, is recorded, not chosen, when you run the task in your own code and ingest the outputs.

Running one

New experiment is one page in four parts:

• Dataset. The latest snapshot is pinned automatically; a run never reads a dataset that is still changing.
• Models, one or more. Each model answers every row with the row's own input (the transcript, or the variables). The first is the baseline the others are compared with. Compare models adds the same setup on another model. A model's Input can instead be a managed prompt and version, compiled with the row's variables, for the prompt-registry case.
• Scorers: the built-in checks (exact match, contains, JSON valid, latency, cost, an LLM judge with a rubric) and your evals: the span-level evals that grade production, run offline on each answer. Run-level and conversation-level evals are listed as not applicable: an experiment cell is one answer to one input.
• Expected: set the dataset's Expected from the models agreeing. See below.

Under Advanced: repetitions, row limit, concurrency.

The row's expected output is held back and fed only to scoring.

Expected from agreement

When several models answer the same row and agree, that answer is a label nobody had to type. The Expected section sets the rule:

Same answer when: the text matches (case, whitespace and JSON key order ignored, no extra model calls) or a model says they mean the same (one judge call per row groups the answers). The judge is the baseline model unless you pick another.

When the run finishes, each winning answer becomes the row's Expected on the live dataset as a derived, confirmed label at weight 0.6, a dataset version named run N agreement is cut, and the run shows N rows got Expected · M left empty: the models disagreed. Rows that already had an Expected are kept unless you choose Overwrite. Rows the models disagreed on stay unlabelled: review them by hand, or run again with a stronger model.

With an agreement rule set, a scorer is optional: the models grade each other.

Scoring

An item passes when every scorer passes. A cell shows the mean of its scorer scores, used for sorting and deltas. The results table is one row per dataset item: Input, Expected, then one column per model with its answer, pass dot and score, so the models read side by side.

Evaluators

Your evals in the Scorers step are the span-level evals that grade production, run offline on each answer with the same graph and version. An eval that needs what a single answer cannot provide (a whole run, a conversation, tool calls, a human reviewer) is listed with the reason and cannot be picked.

Every run is measured against a baseline

So you do not have to ask for that second answer. Every run picks a baseline, by default the most recent completed run on the same dataset, and classifies every scenario against it when the run finishes:

The experiment page leads with that breakdown, regressions expanded, each showing the baseline output beside the new one. The verdict is computed once when the run completes and stored: it is a fact about the run, not something recalculated differently on each page load.

Compare against in the New experiment dialog overrides the default: pick any earlier run, or Nothing to score a run on its own.

What counts as a regression

A score has to move by more than 0.05 to count. Two runs of an identical configuration routinely differ by a couple of hundredths: treating that as a regression would fill the list with noise and train you to ignore it.

The exception is the pass threshold. If a scenario passed at 0.71 and now fails at 0.69, that is a regression even though it moved 0.02: it crossed the line between two different answers to "is this good enough". A tolerance band that swallowed that would be worse than no tolerance at all.

Comparing two runs directly

Select two experiments and Compare to see their aggregates side by side with per-metric deltas. Lower-is-better metrics (contradiction, refusal, cost, latency) colour their deltas accordingly.

Per-row outputs align by scenario id, not by row position, so you are always reading the same test case on both sides. That holds even if the dataset changed between the two runs: rows added since the earlier run simply have nothing to compare against, rather than shifting every row beneath them and quietly turning an unrelated pair into a "regression".

When we refuse to show you a diff

A diff that lines up the wrong rows still looks like a diff. Two cases are called out rather than rendered:

• The runs share no scenarios. Their rows could only be matched by position, which pairs unrelated test cases. The per-item table is withheld entirely: aggregates still show, but no verdict is offered, because any pairing would be a guess.
• More than one thing changed. If a run differs from its baseline in prompt and model and dataset, you get a banner naming each change: "these differ in 3 dimensions: this diff is not attributable to a single change." The numbers are real; they just are not evidence about any one of those changes.

You will also see a note when two runs are configured identically: any difference between them is run-to-run variance, not something you did.

Series

Runs on the same dataset form a series: the set it makes sense to line up, because the dataset is what is held fixed while something else varies. Baselines are chosen from within a series, and run names are prefixed with it.

Names are generated for you unless you type one (support-faq-prompt-refund-policy-v3-2026-08-12T09-52-32Z), carrying the dataset, what was under test, and when. A history of those reads as a history; a history of test, test2, final does not.

Turn results into labels

A run's outputs can become a dataset's Expected. Open the run, choose Save outputs to dataset, and pick the dataset, the rows (all, passed, failed, improved, regressed) and whether to fill empty Expected only. Rows whose cell passed its required scorers are saved as confirmed labels; the rest arrive as proposals to review. This is how a dataset curated from production, with its Expected still empty, becomes labelled data you can export for a fine-tune. See Label with an experiment.

Ingest outputs from your own agent

When your task isn't a single prompt (a multi-step agent, a RAG pipeline, a chain you run in your own code) run it over the dataset yourself and submit the outputs. Trodo scores them against the dataset's ground truth and stores a normal, comparable experiment.

const dataset = 'qa-golden-set';
const outputs = rows.map((row, i) => ({
  itemPosition: i,
  output: myAgent.run(row.input),   // whatever your task produces
}));

await trodo.experiments.ingest({
  dataset,
  name: 'my-agent v3',
  outputs,
  evaluatorIds: ['conciseness-judge'],   // optional: your reusable evaluators
});
// → the created experiment record

dataset = "qa-golden-set"
outputs = [
    {"item_position": i, "output": my_agent.run(row["input"])}
    for i, row in enumerate(rows)
]

trodo.ingest_experiment(
    dataset=dataset,
    name="my-agent v3",
    outputs=outputs,
    evaluator_ids=["conciseness-judge"],  # optional
)
# → the created experiment record

Each output is matched to its dataset row by item_position. Add a judge ({ credential_id, provider, model }) and/or evaluator_ids to control grading. The result is an experiment with task.type = external, sitting alongside your prompt/model runs and comparable to them.

Two optional fields make ingested runs first-class:

• system_version_label on the request: a git sha, a build id, agent v3. This is what makes one agent build distinguishable from the next in a series; without it, two ingests of different builds look identical.
• agent_run_id on an individual output: if you traced the run that produced it, this links the scenario to its trace, so a regression pivots straight to what actually happened instead of being re-run to find out.

Next

• Datasets: build and curate the question set an experiment runs on
• Import files: upload CSV, Excel, JSON, or JSONL
• Curate from production: promote traces and experiment rows
• Playground: spot-check a single case interactively
• Evals: score live production traffic
• Prompt Management: version prompts and pin them by hash