AI Score

The AI Score is how Trodo grades the output of a single LLM node. Every successful LLM span is scored on seven generation-quality metrics, each measuring one way an output can go wrong. A score that crosses its threshold becomes an occurrence, and those occurrences roll up into signals exactly like any other pattern.

Unlike the other pattern families, the AI Score is computed from your input. How you send data to an LLM node — as one blob, or as a chat-message array (system / user / assistant / tool messages, plus a context role for RAG docs) — decides which of the seven Trodo can compute. Send the right data covers this; read it if you want the full score.

The seven scores at a glance

Each score in depth

Contextual grounding

What it means. Blends semantic similarity and entailment between the output and its grounding source — the context messages you passed, or (when there are none) the tool results and prior assistant turns in the transcript. A low score means the answer isn't backed by any source — the classic hallucination shape, where the model is answering from its own memory instead of the material it was given.

How to improve. Retrieve better, more relevant context and pass it as a context message. Instruct the model to answer only from the provided context and to say when it doesn't know. If grounding is low but the answer is correct, your retrieval is probably missing the source the model actually used.

Needs: a grounding source — context, tool, or assistant messages. Skipped when none is sent.

Rule adherence

What it means. The share of mechanical rules in your system messages that the output actually followed. Trodo checks the ones it can verify objectively: "respond in JSON," word/length caps ("under 50 words," "one word"), required phrases, and forbidden phrases. The healthy score is a perfect 1.0 — every checkable rule passed.

How to improve. State constraints explicitly and mechanically so they're checkable and so the model obeys them: name the format, give the word cap as a number, quote required or forbidden phrases. Validate and repair the output (e.g. JSON parse-and-retry) before returning it.

Needs: a system message containing at least one checkable rule. Skipped when there's nothing mechanical to check.

Logical contradiction

What it means. The probability the output contradicts the context or reverses a key fact — a status flipped (active → inactive), a negation added or dropped, or a number from the context swapped for a different one in the output.

How to improve. Remove conflicting or stale material from the context. Tell the model not to contradict the source. Watch numbers especially: don't let the model restate figures it might round or invent — have it quote them.

Needs: a grounding source — context, tool, or assistant messages. Skipped when none is sent.

Trajectory alignment

What it means. Compares the direction from this node's input to its output against the node's own historical average. A drop means this output moved somewhere unusual for this node — a confabulation or drift from how it normally behaves. It's relative to the node's baseline, so it needs history to judge against.

How to improve. Keep each node's prompt and model stable; a sudden drop often lines up with a prompt or model change, or off-distribution inputs. Because it's baseline-relative, a node only starts scoring once it has enough history (a handful of prior runs); brand-new nodes show as skipped until the baseline warms up.

Needs: user messages (or a plain input) and a warmed-up baseline for the node. Skipped on cold start or missing embeddings.

Factual retention

What it means. The share of hard facts in the source — IDs, numbers, money, dates, URLs, proper nouns — that survive into the output. Entities must be mostly retained (≥ 0.70); numbers must be retained perfectly, since a dropped or altered number is usually a real error. Aimed squarely at summarization and extraction.

How to improve. For summaries and extractions, instruct the model to preserve every identifier, number, and date verbatim. Keep the source material in a context message so Trodo can compare against it. If numbers are being reworded or rounded, pin them explicitly in the prompt.

Needs: a source (context, else tool/assistant messages, else the input) with at least two hard facts. Skipped when the source has too few facts to judge.

Semantic echo

What it means. How similar the output is to the input. A very high score means the model barely transformed the prompt — it echoed or lightly paraphrased it instead of doing work. Lower is better here.

How to improve. Make sure the node actually transforms its input — summarize, answer, or reason — rather than restating it. If a node is legitimately a pass-through (e.g. a formatter that intentionally returns the input), disable this pattern for that node in Thresholds rather than fighting the score.

Needs: input and output. Skipped if embeddings are missing.

Refusal score

What it means. How strongly the output reads as a refusal or deflection — "I'm sorry, I can't help with that," "as an AI language model…," or an error/quota message surfaced as the answer — when the request was one the model should have handled. Detected both lexically and by similarity to known refusal shapes.

How to improve. Loosen over-cautious safety instructions so the model stops refusing safe requests. Make sure upstream errors (a 5xx or rate-limit) aren't being returned to the user as the model's answer. Genuine, appropriate refusals of unsafe requests are expected — this score targets refusals of requests the model should answer.

Needs: only the output. Always available.

Send the right data

The AI Score is only as complete as the input you send. Trodo embeds an LLM span's input as a whole and each role separately, so when you send the input as a chat-message array — system / user / assistant / tool messages, plus the context role for RAG docs — each score can reason about the right part. Structure it once — see Structure LLM input for the exact SDK call:

span.setInput([
  { role: 'system', content: systemPrompt },       // rules & role
  { role: 'context', content: retrievedDocs },     // RAG docs (Trodo extension)
  { role: 'user', content: userQuestion },         // the actual user turn(s)
  { role: 'tool', content: toolResult },           // tool output, if any
]);

Which role unlocks which score

Send a plain string instead and Trodo still embeds it as one vector — you'll get Semantic echo, Trajectory, and Refusal. But Rule adherence has no system instructions to parse, and the three grounding scores — grounding, contradiction, and factual retention — have nothing to compare against and will skip. Multiple messages per role are fine: all of a role's messages are combined, in order, into that role's vector.

Which scores matter in which situation

You don't need every score on every node — send the fields that fit what the node does:

Best practices

• Send the same messages you send to the model. The message array is the native shape — no restructuring needed. RAG, tool-augmented, and multi-turn prompts benefit most.
• Put the actual retrieved source in a context message, not a summary of it. Grounding, contradiction, and factual retention all compare the output against exactly what you send here. Without it, tool results and prior assistant turns serve as the grounding source.
• Make constraints mechanical in the system message. "Respond in JSON," "under 50 words," and quoted required/forbidden phrases are what Rule adherence can check.
• Keep user messages to the real user turns. Anchoring on the actual ask (not the whole prompt blob) makes Trajectory and Semantic echo sharper.
• It's non-breaking. The message array is optional — plain strings keep working; you just unlock more scores by sending the transcript. Auto-instrumented spans already arrive in this shape.

Thresholds and tuning

The AI Score patterns live under Generation quality in Settings → Issue thresholds, alongside every other pattern. Each score's pass/fail threshold is fixed (the values in the table above), but you control how many crossings — over how many users, in what window — it takes to fire a signal, and you can disable any score you don't want to track. See Thresholds.

Next

• Patterns — the full failure catalog
• Thresholds — turning crossings into signals
• Structure LLM input — the SDK call