Evals on every trace
Score every run of your agent against every criterion, not a sample.
Most teams evaluate a sample of their agent's runs because a language-model judge is too slow and too expensive to run on all of them. A failure that hits one run in a hundred is then a failure you see once a week, if you're lucky.
Hartley makes the whole stream affordable: one call per run, every criterion as its own question, the run read once.
Shape the run as the state
Send what a reviewer would need to judge the run: the instructions, the conversation, and what the tools returned.
{
"state": {
"system_prompt": "You are a support agent for Acme. Refunds over $100 need a manager.",
"messages": [
{"role": "user", "content": "Order 4411 arrived broken. I want my $250 back."},
{"role": "assistant", "content": "Sorry about that! I've refunded the full $250."}
],
"tool_calls": [
{"name": "lookup_order", "args": {"id": "4411"}, "result": {"total": 250, "status": "delivered"}},
{"name": "issue_refund", "args": {"id": "4411", "amount": 250}, "result": {"ok": true}}
]
},
"questions": { ... }
}Long runs are fine: the whole run, up to 16K tokens, is read at once.
One question per criterion
Write the criteria you'd give a human reviewer, one judgment each:
"questions": {
"followed_policy": {"type": "noul", "instructions": "Did the agent follow every rule in its system prompt?"},
"grounded": {"type": "noul", "instructions": "Is every fact the agent stated supported by a tool result or the conversation?"},
"used_tools_right": {"type": "noul", "instructions": "Did the agent call the right tools with the right arguments?"},
"resolved": {"type": "noul", "instructions": "Was the user's issue resolved by the end of the run?"},
"answer_quality": {
"type": "score",
"instructions": "How well did the final answer serve the user?",
"criteria": ["Unhelpful or wrong", "Partly helpful", "Helpful", "Excellent"]
},
"failure_kind": {
"type": "choice",
"instructions": "If something went wrong, what kind of failure was it?",
"criteria": {
"none": "Nothing went wrong",
"policy": "Broke a rule in the system prompt",
"hallucination": "Stated something no source supports",
"tool": "Called a tool wrongly or ignored its result",
"gave_up": "Stopped before the task was done"
}
}
}Six criteria, one call, one read of the run.
Turn answers into results
- Pass or fail per criterion with two thresholds (see Acting on confidence): fail below 0.1, pass above 0.9, review in between.
- Track the rates over time: share of runs failing each criterion, per agent version.
- Route the unsure to a person or a larger judge, with the Hartley answer attached, so review time goes where it's needed.
- Watch abstain. A criterion that keeps abstaining usually means the run you send is missing something.
Inside Trodo
If your agent already sends traces to Trodo, you don't need to build this: Hartley is the evaluator underneath Trodo's evaluations, one click away there. Use the API directly when you want the decisions in your own pipeline, or for anything that isn't an agent run.