Overview
Signals turn the raw failures in your traces into a short, ranked list of problems worth acting on. Built on patterns, gated by thresholds you control.
A single failed run is noise. The same failure across hundreds of runs is a signal. Trodo continuously watches your traces and product events for known failure patterns, and raises a signal the moment one recurs past a bar you set. You get a short list of what's actually going wrong, instead of a firehose of individual spans to scroll.
The mental model
Three ideas, in order:
Pattern
A named failure mode Trodo knows how to catch — a tool timing out, a hallucination, a rage-click. See Patterns.
Threshold
The bar you set per pattern: how many times, across how many users, in what window. See Thresholds.
Signal
What fires when a pattern clears its threshold — the pattern plus the exact thing acting up, over the window. This is what you investigate.
Every time a pattern matches on a run or span, Trodo records an occurrence. Occurrences on their own aren't signals — one bad run happens. Trodo counts them in a rolling window, and only when the count clears your threshold does it raise a signal.
How a signal forms
Detect
As runs land, Trodo checks each one against every enabled pattern and records the occurrences it finds.
Group
Occurrences are grouped by what's acting up — the tool, the LLM node, or the page — so one root problem is one signal, not thousands of duplicates.
Fire
When a group clears your threshold — enough occurrences, enough distinct users, inside the lookback window — it fires a signal.
Signals group by the entity that's failing (a specific tool, node, or URL), not by the exact error text. A tool timing out after five minutes and the same tool timing out after six collapse into one signal, so you investigate the problem once instead of chasing near-duplicates.
Foundation: patterns
Patterns are the foundation everything else sits on. Trodo ships a catalog of them across five families:
- Tool failure — a tool call broke (errors, timeouts, rate limits, empty results).
- Tool misuse — tools were called wrongly even when they didn't error (loops, storms, bad ordering).
- Response quality — the conversation went sideways (rephrase loops, language mismatch).
- Generation quality — the model's output has a problem (weak grounding, contradiction, dropped facts). These are also surfaced as the AI Score.
- UX rage — friction in the product, from the browser SDK (rage clicks, dead zones, slow loads).
See Patterns for the full catalog and what each one means.
Two ways patterns become signals
Most patterns are event-based: they match on a discrete thing that happened (a 500, a retry storm, a rage click), and fire once enough of them pile up.
Generation-quality patterns are score-based: every successful LLM node output is graded on seven metrics — the AI Score. A score that crosses its threshold counts as an occurrence, and those occurrences roll up into signals the same way. The AI Score is worth understanding on its own, because how you send data to your LLM nodes decides which scores Trodo can compute.
Turning it on
Signals read the data you already capture. There's nothing to configure to start:
Instrument your agent with Observability so runs and spans flow in. For generation-quality scores, send your LLM input as a chat-message array (system / user / assistant / tool, plus context for RAG docs).
(Optional) Add the browser SDK if you want UX-rage signals like rage clicks and slow loads.
Tune your thresholds — or keep the sensible defaults. Detection runs automatically.
Where signals show up
- On a run or span — a Signals panel lists every pattern that matched, grouped by the entity acting up.
- On an LLM node — the AI Score shows all seven generation-quality metrics for that output, each with its value and threshold.
- Per pattern — thresholds and enable/disable live in Settings → Issue thresholds.
Next
- Patterns — the full catalog and what each pattern means
- Thresholds — how a pattern becomes a signal, and how to tune the bar
- AI Score — the seven generation-quality scores, what they mean, and how to improve them