Curate from production
Turn real production spans and experiment results into dataset rows — recover prompt variables from traced calls, link back to the source log, or export experiment failures as a regression set.
The best test cases come from real traffic and real eval runs. Trodo can promote a production span or an experiment's rows into a dataset without hand-copying inputs and outputs.
From a trace span
- Open Agent Runs or Traces and find the run.
- Click into the span detail for the LLM call you want to capture.
- Click Add to dataset.
- Pick an existing dataset or type a new name.
Trodo turns the span into a row:
| Field | Source |
|---|---|
| Input | When the span carries a trodo.prompt.version_hash, Trodo resolves that exact prompt version and recovers the variable values the call was compiled with. Otherwise it stores parsed query / system / context fields. |
| Expected output | The span's recorded output. |
| Origin | A back-link to the source span — click Log in the grid to jump back. |
Prompt-attributed spans recover richer variable bindings via trodo.prompt.version_hash. See Prompt traceability for how version hashes are stamped on spans.
Rows keep a source_trace_ref back-pointer. The Origin column in the dataset grid shows Log for trace-sourced rows.
Prerequisites
You need tracing wired up so production calls appear in the dashboard. For managed prompts, compile inside a tracked agent run so the span is stamped with the prompt version hash.
From an experiment
After running an experiment, open the experiment detail page and click Create dataset from experiment (database icon).
Configure:
| Option | Choices |
|---|---|
| Dataset name | Name for the new dataset (defaults to experiment name + "dataset"). |
| Rows to include | All rows · Failures only (error or composite < 0.70) · Passes only (composite ≥ 0.70) |
| Expected output source | Keep original ground truth from the source dataset · Use experiment model output as expected output |
| Model | When the experiment ran multiple models, pick which model's output to use (defaults to first result per row). |
When to use each mode
| Goal | Filter | Expected source |
|---|---|---|
| Regression set from failures | Failures only | Keep original ground truth |
| Promote model answer as new truth | All or passes | Use experiment output |
| Full copy of experiment inputs | All | Keep original ground truth |
Experiment-exported rows show Exp in the Origin column — click to open the source experiment.
Compare the two sources
| Trace span | Experiment | |
|---|---|---|
| Best for | A specific production failure with rich prompt context | Batch-exporting eval results |
| Input | Recovered prompt variables or parsed span fields | Original dataset input |
| Expected | Span output | Original ground truth or model output |
| Origin link | Log (span) | Exp (experiment) |
After curating
- Open the new dataset and review rows in the dashboard editor.
- Add tags (e.g.
regression,prod-incident-2026-03) for filtering. - Click Save, then optionally create a snapshot.
- Run an experiment against the curated set to verify a fix.
Next
- Edit in dashboard — clean up curated rows
- Prompt traceability — how spans record prompt versions
- Get started with observability — wire tracing first
- Experiments — run and compare against your curated set
Ingest from SDK
Append dataset rows from Node.js or Python — auto-create datasets by name, push golden sets from CI, or capture production inputs into a reusable test set.
Experiments
Run a task across every row of a dataset, score each result, and store a reproducible, comparable record. Test a prompt version, compare models, attach your production evaluators, or ingest outputs your own agent produced.