Experiments

An experiment runs a task across every row of a dataset and scores each result, producing a reproducible record you can compare against later runs. It's offline evaluation — the loop where you catch a regression before your users do.

Open Experiments in the sidebar, or reach it from a dataset or prompt via Evaluate → Run in Experiment.

The task is a run-time choice

The dataset and the task are chosen independently. A dataset carries named inputs and ground truth; what you run against it is decided per experiment. There are two places a task can come from:

• A prompt — pick a prompt and a version. The experiment compiles that exact version with each row's variables and sends it to your model(s).
• Raw dataset input — no prompt. The row already carries its whole input, so it goes to the model as-is. Use this when the text you want to test is already in the dataset. If the row carries messages, the conversation is sent as-is: the last user turn is the question, earlier turns are context.

The resolved prompt is pinned by its immutable version_hash, so an experiment always records the exact version that ran — even if you later move the production label to a different version. Reproducibility is the whole point.

A third source, external, is recorded — not chosen — when you run the task in your own code and ingest the outputs.

Running one

New experiment asks for:

• Dataset — and optionally a snapshot to pin, instead of the live draft.
• Task — A prompt (then a version, defaulting to production), or Raw dataset input.
• Models — one or more, each a linked provider + model + temperature. Add several to compare gpt-4o vs claude-sonnet vs a cheaper model on the same rows. Models come from the providers you connect under Settings → Integrations (your own keys).
• Compare against — the run to judge this one against; defaults to the most recent on this dataset. See baselines.
• Evaluators (optional) — attach reusable evaluators. Ones attached to the dataset are pre-selected.
• Stability runs — run each row N times to measure agreement.
• Deep eval — add NLI-based contradiction / entailment checks.

Each row's input binds to the prompt's {{variables}} by name; the row's expected output is held back and fed only to scoring.

Scoring

Results use the same engine as the playground — relevance, correctness, groundedness, entity fidelity, adherence, contradiction, refusal, stability, and a composite alignment score. A row's expected output, when present, anchors the correctness judgement (does the answer match the reference?).

Each experiment persists a per-model aggregate: the mean of each dimension, pass rate (alignment ≥ 0.70), and average cost and latency — plus the full per-row table of outputs and scores.

Evaluators — the same ones that grade production

Beyond the built-in scores, you can attach your reusable evaluators — the exact LLM-judge / code evaluators that grade your live traffic — via the multiselect in the New experiment dialog. They run offline over the experiment's rows and appear as an Evaluators · offline aggregate plus a per-row column, scored identically to production. One definition of "good", online and offline.

Evaluators can also belong to the dataset. Attach them once and every run of that dataset inherits them, pre-selected and removable for a single run. This is the recommended setup: how a dataset should be graded is a property of the dataset, and an experiment that nobody remembered to attach a scorer to produces outputs rather than a verdict.

Every run is measured against a baseline

So you do not have to ask for that second answer. Every run picks a baseline — by default the most recent completed run on the same dataset — and classifies every scenario against it when the run finishes:

The experiment page leads with that breakdown, regressions expanded, each showing the baseline output beside the new one. The verdict is computed once when the run completes and stored — it is a fact about the run, not something recalculated differently on each page load.

Compare against in the New experiment dialog overrides the default: pick any earlier run, or Nothing to score a run on its own.

What counts as a regression

A score has to move by more than 0.05 to count. Two runs of an identical configuration routinely differ by a couple of hundredths — treating that as a regression would fill the list with noise and train you to ignore it.

The exception is the pass threshold. If a scenario passed at 0.71 and now fails at 0.69, that is a regression even though it moved 0.02: it crossed the line between two different answers to "is this good enough". A tolerance band that swallowed that would be worse than no tolerance at all.

Comparing two runs directly

Select two experiments and Compare to see their aggregates side by side with per-metric deltas. Lower-is-better metrics (contradiction, refusal, cost, latency) colour their deltas accordingly.

Per-row outputs align by scenario id, not by row position, so you are always reading the same test case on both sides. That holds even if the dataset changed between the two runs: rows added since the earlier run simply have nothing to compare against, rather than shifting every row beneath them and quietly turning an unrelated pair into a "regression".

When we refuse to show you a diff

A diff that lines up the wrong rows still looks like a diff. Two cases are called out rather than rendered:

• The runs share no scenarios. Their rows could only be matched by position, which pairs unrelated test cases. The per-item table is withheld entirely — aggregates still show, but no verdict is offered, because any pairing would be a guess.
• More than one thing changed. If a run differs from its baseline in prompt and model and dataset, you get a banner naming each change: "these differ in 3 dimensions — this diff is not attributable to a single change." The numbers are real; they just are not evidence about any one of those changes.

You will also see a note when two runs are configured identically — then any difference between them is run-to-run variance, not something you did.

Series

Runs on the same dataset form a series: the set it makes sense to line up, because the dataset is what is held fixed while something else varies. Baselines are chosen from within a series, and run names are prefixed with it.

Names are generated for you unless you type one — support-faq-prompt-refund-policy-v3-2026-08-12T09-52-32Z — carrying the dataset, what was under test, and when. A history of those reads as a history; a history of test, test2, final does not.

Ingest outputs from your own agent

When your task isn't a single prompt — a multi-step agent, a RAG pipeline, a chain you run in your own code — run it over the dataset yourself and submit the outputs. Trodo scores them against the dataset's ground truth and stores a normal, comparable experiment.

const dataset = 'qa-golden-set';
const outputs = rows.map((row, i) => ({
  itemPosition: i,
  output: myAgent.run(row.input),   // whatever your task produces
}));

await trodo.experiments.ingest({
  dataset,
  name: 'my-agent v3',
  outputs,
  evaluatorIds: ['conciseness-judge'],   // optional: your reusable evaluators
});
// → the created experiment record

dataset = "qa-golden-set"
outputs = [
    {"item_position": i, "output": my_agent.run(row["input"])}
    for i, row in enumerate(rows)
]

trodo.ingest_experiment(
    dataset=dataset,
    name="my-agent v3",
    outputs=outputs,
    evaluator_ids=["conciseness-judge"],  # optional
)
# → the created experiment record

Each output is matched to its dataset row by item_position. Add a judge ({ credential_id, provider, model }) and/or evaluator_ids to control grading. The result is an experiment with task.type = external, sitting alongside your prompt/model runs and comparable to them.

Two optional fields make ingested runs first-class:

• system_version_label on the request — a git sha, a build id, agent v3. This is what makes one agent build distinguishable from the next in a series; without it, two ingests of different builds look identical.
• agent_run_id on an individual output — if you traced the run that produced it, this links the scenario to its trace, so a regression pivots straight to what actually happened instead of being re-run to find out.

Next

• Datasets — build and curate the question set an experiment runs on
• Import files — upload CSV, Excel, JSON, or JSONL
• Curate from production — promote traces and experiment rows
• Playground — spot-check a single case interactively
• Evaluations — author the reusable evaluators experiments can attach
• Prompt Management — version prompts and pin them by hash