LLM judge

An LLM judge sends your prompt, filled in from the trace, to a model you choose, and gets back exactly the fields the step declares. Use it for what needs judgement: whether an answer is relevant, complete, on-brand, in scope, safe; which of five failure types it is.

The model

Pick a Provider and Model from the providers you added in Settings → Models: OpenAI, Anthropic, Gemini, Azure OpenAI and OpenAI-compatible endpoints. The judge runs on your key: your provider bills you for its calls, and nothing else in an eval does. Params sets temperature (0 by default, judges should be repeatable), max tokens and top-p.

Each judge step picks its own model, so one eval can use a small, fast model for a first opinion and a stronger one only for the cases that reach it.

The prompt

Write what you want decided, and mark where trace data goes with {{…}}:

An assistant answered using tool results.

Question:
{{run.input}}

Tool results:
{{steps.gather.evidence}}

Answer:
{{run.output}}

Score 0 to 1 how well every figure and claim in the answer is supported by the tool results
(1 = all supported, 0 = invented).

You don't write the answer format. Trodo appends an instruction to reply with exactly the step's declared fields as JSON (a number between 0 and 1, one of your options, true or false) and checks what comes back. A malformed reply gets one corrective retry; a second one errors the result.

An optional System prompt sits behind a disclosure under the prompt.

Variables

Click a field in the list under the prompt to insert it. A variable that is empty on a trace renders as nothing; if no variable resolves, the result is errored rather than judged on an empty prompt.

What it returns

Declare the output on Output & routing:

The judge also returns its reasoning: shown on every result's path, and readable by later steps as {{steps.<judge>.reasoning}}.

Writing prompts that decide well

• Give it the evidence, not the haystack. Gather what matters in a Python step (the tool outputs, the retrieved passages, the expected answer) and pass exactly that. A judge given a whole trace does worse and costs more.
• Define the scale. "1 = every claim supported, 0 = invented" beats "rate the quality".
• Prefer categories to fine scores when you'll act on the category: wrong / unhelpful / tone / other is a report, 0.63 is not.
• Leave the easy cases out. If a screen has already passed the clean answers, say so: "This answer's figures did not all appear in the tool output."
• Send the unsure band to a person. Route a middle range of scores to Human review; the grades become labels, and self-improving learns from them.

Cost and speed

A judge step is usually the slowest and the only billed-by-your-provider step in an eval. Its tokens and cost are recorded on every result, summed in Provider spend on Analysis, and estimated before you re-score history. Testing a judge in the editor calls your provider for real.

When it errors