Question types
noul, choice, score and extract.
Four kinds of question cover every decision: yes or no, pick one, rate it, or point at a value in the state. Each comes back as numbers or as a span of your own input, never generated text.
type | Asks | criteria | Answer field |
|---|---|---|---|
noul | a yes/no proposition | optional {"yes": "…", "no": "…"} describing each case | noul: P(yes) |
choice | pick one of N (2–255) | a list of names, or {name: description} (descriptions help) | choice and probabilities |
score | an ordered rating (2–10 levels) | a list of level descriptions, lowest first | score: the expected level (0-based) and probabilities |
extract | a value stated in the state | none; extract.kind narrows the candidates | present, value, span (character offsets into the state) and candidates |
noul
{"type": "noul", "instructions": "Did the agent follow its system prompt?"}choice
{
"type": "choice",
"instructions": "Which tool should the agent call next?",
"criteria": {
"search_orders": "Look up the user's orders",
"issue_refund": "Refund a payment",
"handoff": "Hand the conversation to a person"
}
}Option names must be unique.
score
{
"type": "score",
"instructions": "How helpful was the agent's last reply?",
"criteria": ["Not helpful", "Somewhat helpful", "Helpful", "Exceptionally helpful"]
}extract
{
"type": "extract",
"instructions": "What budget did the lead state?",
"extract": {"kind": "amount"}
}The answer is always a span of the state the model pointed at, never a generated value, so an extracted amount, date, phone number, email, name or phrase is exactly what the input said:
{"type": "extract", "present": true, "value": "2.5 lakh", "span": [118, 126], "confidence": 0.82,
"candidates": [{"value": "2.5 lakh", "span": [118, 126], "probability": 0.84}, {"value": "500", "span": [96, 99], "probability": 0.09}],
"abstain": 0.05}extract.kind is one of any (default), number, amount (money, with lakh / crore / k / %), date, phone, email,
name or text (short phrases); extract.max_candidates (1–128, default 48) caps how many spans the model considers.
When nothing in the state fits, present is false and abstain carries the mass on "not stated". Values that are not
literally in the text (a computed total, a normalised date) are for your code to derive from the span.
Abstain
abstain is reserved: it is added to every question automatically and reported separately, as the mass the model puts on "the state doesn't say". Turn it off with options.abstain: false.