Curate from production
Turn real spans, runs, conversations and eval results into dataset rows: pick them in a table, fill a dataset from a filter, or keep adding new matches every 15 minutes.
The best test cases come from real traffic. A row curated from production keeps what actually happened: the Input the system was given, the Output it produced, and the Evaluations that scored it. Expected starts empty. What the system said is not automatically what it should have said; that is a label, and you decide it.
What a curated row holds
| Field | Source |
|---|---|
| Input | The unit's transcript, one column: every turn with its role. For a span with a trodo.prompt.version_hash, also the prompt variables it was compiled with (shown under the transcript in the row). |
| Output | The whole assistant message the span, run or conversation produced: text, tool calls, structured JSON. |
| Evaluations | A snapshot of every verdict on the unit: eval outcomes, thumbs, the correction and reason when someone marked it wrong, and per-step labels. Refresh evaluations re-reads them later. |
| Expected | Empty and unlabelled, unless you ask for passing outputs to be accepted (see below). |
| Source | A back-link to the span, run or conversation. A unit is in a dataset at most once: adding it again is skipped. |
The unit is what a row is about: one LLM span, one agent run, or one conversation.
Pick rows in a table
- Open Traces (spans or runs) or an eval's Results.
- Tick the rows you want. The selection bar shows Add to dataset…
- Pick a dataset or type a new name.
To take everything the page's filter matches instead, click Dataset from filter in the toolbar: it opens the From spans (or From runs) template with the filter already filled in.
Adding a selection is free. A result you mark wrong on an eval's Results page carries your correction and reason into the row's Evaluations, which is what a proposal works from.
Fill a dataset from a filter
New dataset offers a source: From spans, From runs, From conversations or From eval results. Each takes the same filter its page uses, plus a time window:
| Source | Filter |
|---|---|
| Spans, Runs | The Traces filter chips, a time window, and three evaluation filters: Evaluated by, Verdict (failed / passed) and Marked wrong / Marked right. |
| Conversations | Agents and a time window. |
| Eval results | Evals, Outcomes, a time window, and Rows are: what the eval scored (one row per span, run or conversation, as above) or the verdicts (one row per verdict, see below). |
Verdict rows
A verdict row is the training set for the eval itself, not for the agent. Its columns mean something different:
| Field | Content |
|---|---|
| Input | What the eval saw: the unit's transcript with the output it judged as the last turn. |
| Output | What the eval said: Pass or Fail, the reason code, the path, and each step's answer. |
| Expected | The person's correction, when there is one. A thumbs-down on the verdict becomes { passed, steps } (the corrected verdict and any step marked should have been). A thumbs-up makes the verdict its own Expected. Both are explicit labels at weight 1.0, by the person who voted. No vote: unlabelled. |
| Source | The eval and version, and the unit it scored. Open verdict goes to the eval's results with that verdict pinned. |
Refresh evaluations re-reads the votes and rewrites Expected, unless a person typed one by hand. On an eval's Results page, Add to dataset… offers the same choice: what it scored or the verdicts.
The dialog previews how many units match before anything is written. A window is always bounded: with none, the last 7 days are used. Options: a limit (up to 10,000 rows) and a sample percentage.
This is a feed: a one-time copy with provenance. The dataset's Feeds panel lists each feed with what it matched, wrote and skipped, and lets you retry a failed one. To add a feed to an existing dataset, use Add rows from a filter… in that panel.
Keep adding matching rows
Tick Keep adding matching rows and Trodo also creates an agent: every 15 minutes a Find dataset units step runs the same filter over what arrived since the last run, and an Add rows to a dataset step writes the matches. It is an ordinary agent: you can see its runs, pause it, change the schedule, or add steps (a filter, a Slack message) in between. See Nodes.
A dataset with an active agent reads Live in the Datasets list and in its header. The Feeds panel shows each agent's last run and the rows it has added, with Pause and Resume.
To set one up for a dataset you already have, click Automate in the dataset's header. It opens the Agents templates on the two dataset templates, with the dataset filled in: Keep a dataset fed from a filter, and Collect every failed verdict into a dataset.
What it costs
Every row a feed or an agent writes is one Execution from your plan's pool. An agent run that finds nothing new costs nothing. Before writing, the feed checks your remaining allowance; if the rows would not fit, nothing is written and the feed says why. Picking rows in a table, refreshing evaluations and exporting are free. See Plans and billing.
Accept passing outputs as Expected
When you add units or create a feed you can tick Accept a passing output as Expected. A unit whose every verdict passed, and that nobody marked wrong, gets its Output copied into Expected as an implicit label. Units that failed or were never scored stay unlabelled. Implicit labels are the weakest kind; confirm them before you train on them.
From an experiment
After running an experiment, open it and click Create dataset from experiment (database icon).
| Option | Choices |
|---|---|
| Dataset name | Defaults to the experiment name + "dataset". |
| Rows to include | All rows · Failures only (error or composite < 0.70) · Passes only (composite ≥ 0.70) |
| Expected output source | Keep the original ground truth · Use the experiment model's output |
| Model | When the experiment ran several models, which model's output to use. |
Experiment-exported rows show Exp in the Source column.
After curating
- Open the dataset. The Label column shows what still needs an Expected.
- Label the rows: write Expected, accept the Output, or ask your model to propose one.
- Save a snapshot before you run an experiment against it.
- Export the confirmed rows for a fine-tune.
Next
- Labels: turn curated rows into ground truth
- Export: JSONL for training and offline evals
- Get started with observability: wire tracing first
- Experiments: run and compare against your curated set
Ingest from SDK
Append dataset rows from Node.js or Python: auto-create datasets by name, push golden sets from CI, or capture production inputs into a reusable test set.
Labels
How a row's Expected becomes ground truth: typed by a person, accepted from a passing output, or proposed by your own model and confirmed. Every label records where it came from and how much to trust it.