Acting on confidence

Thresholds, cascades and abstain: automate what Hartley is sure about, and send the rest on.

Hartley's answers are calibrated, so the probability is something you can build on. The pattern is always the same: act on the confident answers, route the rest.

Accept automatically at confidence ≥ 0.85

0.990.980.980.970.960.950.950.930.910.900.880.840.800.740.690.610.550.420.330.18

Accepted automatically11

Sent to a person or a larger model9

Twenty example answers, most confident first. Raise the line and fewer answers are automated, each more likely right; lower it and more are automated. Because the probabilities are calibrated, the line means what it says.

Pick the number to act on

Question type	Act on	Example
`noul`	`noul`, the probability of yes	`if a.noul >= 0.9: ...`
`choice`	`probabilities[option]`, or `confidence` for "is it sure about any option?"	`if a.probabilities["escalate"] >= 0.8: ...`
`score`	`score`, the expected level, and `confidence`	`if a.score <= 1 and a.confidence >= 0.6: ...`
any	`abstain`, the "not enough evidence" share	`if a.abstain >= 0.3: ...`

Two thresholds, three outcomes

For a yes or no decision, use two lines rather than one. Above the top line, act as if yes; below the bottom line, act as if no; in between, Hartley isn't sure, so ask someone who can be.

def decide(a, hi=0.9, lo=0.1):
    if a.get("abstain", 0) >= 0.3:
        return "needs_evidence"      # the state doesn't say: fix the input or ask a person
    if a["noul"] >= hi:
        return "yes"
    if a["noul"] <= lo:
        return "no"
    return "review"                  # Hartley is unsure: a person or a larger model decides

Widen the gap between the lines when a wrong automatic decision is expensive; narrow it when review is the expensive part.

Cascades

The cheapest reliable system is usually a cascade: Hartley decides everything it is confident about, and only the remainder goes to a heavier step.

Hartley on everything. Every run, every ticket, every record.
A larger model on the unsure share. Send only the answers between your thresholds to a frontier model, with the same question.
A person on what's left. The cases neither is sure about, or the ones that matter most.

Because the probabilities are calibrated, you can choose the thresholds from the cost of each mistake and the cost of each review, and they keep meaning the same thing as your traffic grows.

Choosing the lines

Start conservative (hi = 0.95, lo = 0.05), run Hartley alongside your current process for a few days, and look at the answers between the lines: how many there are, and how often the confident ones were right. Then move the lines until the review queue is the size you can afford.

Calibration holds on average, across many answers. A single answer at 0.9 can still be wrong; that's the 10%.

What to do with abstain

A high abstain means "the state doesn't contain the evidence", not "no". Common causes:

The evidence wasn't sent. The tool result, the earlier turn or the policy is missing from the state. Fix the input.
The question doesn't apply. "Did the agent apply the discount correctly?" on a run with no discount. Skip it, or ask whether it applies first.
It really is ambiguous. Send it to a person.

Don't fold abstain into "no". For evals especially, "we can't tell from this trace" is a finding of its own.