Evaluator types
LLM judge, code, human, and composite evaluators, and when to reach for each.
Trodo has four evaluator kinds. They all share the same result shape (score, passed, reasoning), so you can mix them freely on the same runs.
Which to use
The quick version: use code when the answer is objective, LLM judge when it takes judgement, human when you need ground truth, and composite to combine them.
| If the question is... | Use | Why |
|---|---|---|
| "Does the output match an exact rule?" | Code | Deterministic, free, instant, and never drifts. |
| "Is the answer actually good?" | LLM judge | Captures subjective quality that rules can't express. |
| "What's the real correct label?" | Human | A person is the ground truth for accuracy and edge cases. |
| "Did it pass all our checks?" | Composite | Rolls several evaluators into one gate. |
A common setup: a few cheap code checks on every run, one LLM judge for quality, sampled, and a small human pass on a slice for ground truth.
LLM judge
An LLM scores the run against a rubric you write. It uses your own model: add one once on the Models page (LLM judges run on OpenAI, Anthropic, or Gemini), then pick it here. You set a system prompt and a user template with {{...}} placeholders that render against the run, and choose how to parse the model's reply (JSON, a bare number, or a pass/fail verdict).
When to use it. For quality you can describe but not compute: helpfulness, tone, faithfulness to the retrieved context, instruction-following, hallucination checks. Reach for it whenever "good" is a matter of judgement.
Why. A judge expresses nuanced criteria in plain language and scales to every run, which a person can't. Trade-off: it costs money per run and can vary, so write a tight rubric, pick a return type with a clear passing range, and sample rather than scoring 100% if volume is high.
Code (Python / TypeScript)
A deterministic evaluator you write as code. It defines an evaluate(event) function that returns a score and reasoning.
- Python runs in an isolated subprocess.
- TypeScript / JavaScript runs in a sandbox with no network,
require, orprocessaccess.
When to use it. For objective, rule-based checks: regex or keyword matches, JSON-schema or format validation, length and latency limits, "did it call the right tool", numeric tolerances.
Why. It's deterministic, free, and instant, and it never drifts, so it's the right default for anything you can express as a rule. Trade-off: it can only check what you can code, so it won't judge subjective quality.
Human
Manual grading. Matching runs are enqueued into a human queue; a person opens the queue and grades each item (stars, thumbs, or a category, derived from your return type). The grade is written back as a normal eval result.
When to use it. For ground-truth labelling, accuracy on hard or ambiguous cases, and spot checks where you don't yet trust an automated judge.
Why. People are the source of truth, and human grades are also how you sanity-check an LLM judge before relying on it. Trade-off: it's slow and doesn't scale, so sample a slice rather than queuing every run.
Composite
A composite evaluator aggregates the results of other evaluators on the same run. Aggregation is average, min, max, or weighted average for numeric scores, or all pass / any pass for boolean scores. Composites run after their sources so the inputs exist.
When to use it. When "good" means several checks together, for example "passes the format check and scores at least 7 on the quality judge", or a single weighted quality score from several judges.
Why. It turns a set of evaluators into one number or one gate you can track and alert on, instead of watching each separately.
A run feedback signal from trodo.feedback() (see Feedback) is available to evaluators as input, so a judge can factor in what the end user thought.