Overview
Score the quality of your agent runs and spans with LLM-as-judge, code, human, and composite evaluators, live or on history.
Evaluations score your agent's output. You define what "good" means, and Trodo runs that judgement on every matching run or span, live as data arrives or in a backfill over history.
Scoring is fully custom. There are no fixed built-in metrics like "correctness" or "safety"; you describe what each evaluator measures with a prompt, code, or grading instructions.
Evaluator types
LLM judge
An LLM scores the run against a rubric you write, using your own model key.
Code
Deterministic Python or TypeScript that returns a score and reasoning.
Human
Manual grading: items land in a queue for a person to score.
Composite
Combine other evaluators into one score with averaging or pass rules.
See Evaluator types for each in detail.
What a result looks like
Every evaluator, whatever its type, produces the same result shape:
- A score whose form depends on the return type: a number, a boolean, or a category string.
- A passed flag, computed against a passing range you set (null when the eval errored, so "failed" and "errored" stay distinct).
- Reasoning: a short explanation from the judge, code, or human.
- Cost and duration for the evaluation itself (cost applies to LLM judges and is an estimate).
Live or on history
- Live: when a run or span is ingested, enabled evaluators score it automatically.
- Backfill: run an evaluator over historical runs or spans.
- Test: try an evaluator against a single run or span before you enable it.
See Running evaluators.