Overview

A dataset is a fixed set of test cases you re-run as you iterate. Where the playground answers "how does this prompt do on this input?", a dataset is the question set behind "how does it do across all my cases, and did my change make it better or worse?".

Open Datasets in the sidebar. A dataset is not owned by a prompt: it's a reusable question set that any prompt (or any task) can run against. That decoupling is the point: the same qa-golden-set can grade three different prompts, or the same prompt across two datasets.

What a dataset row is

Every row is a test case:

The table shows Input first, then Output, Evaluations, Expected with its Label, Source, Tags, and Metadata, so it's always clear what feeds the model versus what only grades it. The Input cell shows the last user turn with the roles present, or the variables; open the row to read the whole transcript. Expected output is never passed to the task as a variable: that would be leaking the answer.

Why a scenario keeps its identity

Every row gets a scenario id the moment it is created, and keeps it for life. That id is what makes two experiments comparable: when you diff last week's run against today's, results are matched by scenario id, not by row position. Match by position instead and inserting one row at the top shifts every comparison by one, silently, with the numbers still rendering.

So editing a dataset is safe:

• Editing a row updates it in place. The id is unchanged, and every past result that scored it still points at it.
• Removing a row archives it. It leaves the working set but remains resolvable, so an experiment that ran against it is still readable.
• Adding a row mints a new id. Runs from before it existed simply have no result for that scenario, which a comparison reports as new rather than pretending it regressed.

Open any row to see its id in the panel header: click to copy it. You will see the same id on that row's results inside an experiment.

When a scenario is a conversation

Some systems take a question; others take a conversation. A row can hold messages instead of flat input:

{
  "messages": [
    { "role": "system",    "content": "Be concise." },
    { "role": "user",      "content": "I bought a jacket last week." },
    { "role": "assistant", "content": "Thanks, noted." },
    { "role": "user",      "content": "Can I still return it?" }
  ]
}

When the experiment runs without a managed prompt, the last user turn is the question, a system turn becomes the system instruction, and the turns in between are carried as context, so the model answers the final question knowing what came before.

Expected output can be a rubric

Expected output is optional, and it does not have to be a single correct string. Many scenarios are graded against criteria rather than an exact answer, the PRD calls these reference-free. A structured expected output is stored and passed to evaluators as an object, not flattened to text:

{
  "sample_reply": "Refunds are accepted within 30 days of purchase.",
  "must_mention": ["30 days"],
  "should_defer": false
}

Use a plain string when there is one right answer; use an object when the grader needs more than that; leave it empty when a reference-free evaluator (valid JSON, not toxic, stayed on topic) is doing the judging.

How variables bind (datasets aren't tied to a prompt)

Because a dataset isn't owned by a prompt, matching happens by name, at run time. When you run an experiment you pick the prompt, and each input key binds to the prompt's declared {{variable}} of the same name:

• a key that matches a declared variable → bound (and coerced to that variable's type);
• a key the prompt doesn't declare → ignored (so extra columns never reach the model);
• a declared variable with no matching key → renders empty.

That's what makes one dataset reusable across prompts: it just carries named inputs plus ground truth, and each prompt decides which names it consumes.

Ways to build a dataset

Save persists your changes: edited rows are updated in place and keep their scenario id, new rows get one, and rows you removed are archived rather than deleted. Snapshots freeze the current rows as a numbered, immutable version.

Why snapshots

A dataset is a live, editable draft. A snapshot is a frozen version of it. When you run an experiment you can pin it to a snapshot, so "experiment #4 scored 0.92" stays reproducible even after you later add or edit rows. Without it, comparing two experiments run weeks apart is meaningless: the test set moved underneath them. Snapshots are optional per run (you can run against the live draft); reach for one when you want a stable baseline.

How this dataset is graded

A dataset can carry its own evaluators. Attach them once and every experiment run against this dataset inherits them, pre-selected in the New experiment dialog, and removable for one run if you need to.

This is worth setting up. Scoring chosen per run means a run created in a hurry has none, and a run with no scorers produces outputs rather than a verdict. How a test set should be judged is a property of the test set, not of whoever happened to launch the run.

Runs on this dataset form a series

Every experiment on a dataset joins that dataset's series, the set of runs it makes sense to line up, because the dataset is what is being held fixed while a prompt, a model, or an agent build varies.

That is what lets a new run pick a baseline automatically (the most recent completed run) and report which scenarios improved or regressed against it, without you having to open a comparison. See every run is measured against a baseline.

Run it, Evaluate

From a dataset (or a prompt) click Evaluate:

• Run in Experiment: batch every row through a prompt + model(s), scored and comparable. This is offline evaluation; see Experiments.
• Run in Playground: open the interactive playground prefilled with this dataset (or prompt) to spot-check a single case.

Whichever object you launch from is prefilled; you only supply the other half.

Next

• Import files, CSV, Excel, JSON, JSONL
• Edit in dashboard: manual row editing
• Ingest from SDK: append from code
• Curate from production: traces, feeds and experiments
• Labels: label status, proposals and step labels
• Export: the labelled JSONL
• Experiments: run a prompt + models across a dataset
• Evals: scoring your live production traffic