Running
Score live ingestion automatically, backfill over history, or test a single run.
An evaluator can run three ways.
Live (online)
When a run or span is ingested, Trodo checks every enabled evaluator for that target. Matching events are queued and scored automatically: filter match, then sampling, then a de-dupe by evaluator, version, and target, then the score is written. Composites are scored last so their source results already exist.
This is the default mode. Enable an evaluator and it scores new data from then on.
Backfill (offline)
Run an evaluator over historical runs or spans, for example after you create a new rubric. Backfill scans a time window and scores past events, skipping any already evaluated. Human evaluators enqueue grading tasks instead of writing scores directly.
Test
Run an evaluator against a single run or span, bypassing the filter and sampling, and always get a verdict back. Use it to validate a rubric before enabling it.
Evaluations always run against real ingested traces (runs and spans), live or backfilled. There is no separate dataset or test-case store today.