Writing good questions

Hartley answers the question you asked, about the state you sent. Most bad answers trace back to one of those two. Six habits fix nearly all of them.

1. Ask one thing per question

A compound question forces Hartley to merge two judgments into one number, and you can't tell which half drove it.

// one number for two things
"ok": {"type": "noul", "instructions": "Was the agent polite and did it resolve the issue?"}

// two numbers, about the same price: the state is read once
"polite":   {"type": "noul", "instructions": "Was the agent polite throughout?"},
"resolved": {"type": "noul", "instructions": "Was the user's issue resolved by the end?"}

Splitting is nearly free: questions share the read of the state, so five atomic questions cost about the same as one.

2. Say what yes means

Name the thing you're judging and where to look for it. "Did the agent follow its system prompt?" works because the system prompt is in the state. "Was this good?" leaves Hartley to guess what good is.

For a noul, you can describe both cases in criteria:

{
  "type": "noul",
  "instructions": "Did the agent promise something it can't do?",
  "criteria": {
    "yes": "The agent committed to an action, refund or timeline outside its tools or policy",
    "no": "Every commitment is something its tools or policy allow"
  }
}

3. Describe the options

For choice and score, give each option a description, not just a name. Hartley reads them alongside the state, and a description turns a label into a definition.

{
  "type": "score",
  "instructions": "How well did the agent answer the user's question?",
  "criteria": [
    "Didn't answer it, or answered a different question",
    "Partly answered; key details missing or wrong",
    "Answered correctly, with everything the user needed",
    "Answered correctly and anticipated the obvious follow-up"
  ]
}

Levels go lowest first. Keep options mutually exclusive: if two could both be right, merge them or split the question.

4. Put the evidence in the state

Hartley sees only what you send. If the question is about whether a tool result was used correctly, the tool result has to be in the state. If it's about following a policy, include the policy. A high abstain is often Hartley telling you the evidence is missing.

Structure helps. Send JSON with named parts rather than one long string, so the question can point at them:

"state": {
  "system_prompt": "...",
  "messages": [ ... ],
  "tool_calls": [ {"name": "lookup_order", "args": {...}, "result": {...}} ]
}

5. Keep "not enough evidence" on

abstain is on by default, and for evals and guardrails you should leave it on. It separates "Hartley thinks no" from "the state doesn't say", which are very different things to act on. Turn it off (options.abstain: false) only when you need a forced choice, for example when routing must always pick a branch.

6. Decide on confidence, not on the top option

choice: "escalate" at 0.41 and at 0.97 are not the same decision. Act on the probability, set a threshold, and send the unsure answers somewhere useful. Acting on confidence shows how.

A checklist

• One judgment per question.
• The question names what to look at, and that thing is in the state.
• Every option has a description; levels are lowest first; options don't overlap.
• abstain on, unless a forced choice is the point.
• Your code reads the probability, not just the top option.