Overview
Alerts watch one number about your runs or spans — latency, cost, error rate, tokens, tool calls — and notify Slack or a webhook the moment it crosses a threshold you set.
An alert watches one number about your agents and tells you when it crosses a line you drew. Latency past eight seconds. Error rate above 5%. Spend over $50 an hour. Trodo recomputes that number every minute over a rolling window, and the first time it breaches, you hear about it in Slack or on a webhook.
Alerts are the half of monitoring that you specify. Signals are the half Trodo finds on its own.
Alerts or Signals?
Both answer "watch my agents for me", and they sit next to each other in the sidebar for that reason. They are not the same job.
| You get told when | You define | |
|---|---|---|
| Alert | A number you chose crosses a line you drew. | The measure, the threshold, the window. It fires on your terms, whether or not Trodo thinks it's interesting. |
| Signal | A failure pattern Trodo knows about recurs. | Only how much recurrence counts. Trodo brings the patterns — retry storms, rage clicks, weak grounding. |
Use an alert when you already know what "too far" looks like: an SLA, a budget, a latency target your team agreed on. Use signals to catch the failures you would never have written a rule for.
The mental model
Four ideas, in order:
Scope
Run or span. A run is a whole agent execution; a span is one step inside it — an LLM call, a tool call, a retrieval. See Runs and spans.
Measure
The number: duration, cost, error rate, tokens, tool calls, users, satisfaction. Each comes with the aggregations that make sense for it — a count has none, a duration has p95.
Filter
Which rows count. Without one the alert watches everything; with one it watches only checkout-agent, or only spans on gpt-4o.
Condition
The threshold and the window: above 8 s, over the last 15 minutes. The window is rolling — it always ends now.
Put together, an alert reads as one sentence: "p95 duration of spans on charge_card, above 5 s, over the last 15 minutes."
How an alert is evaluated
Every minute
Trodo reads the rows in your window — the last 15 minutes, the last 24 hours, whatever you set — and keeps only the ones your filter matches.
Reduce to one number
Those rows become a single value using your aggregation: a count, a rate, an average, a p95.
Compare, and notify on the crossing
If the value breaches your threshold and it wasn't breaching before, the alert fires and the notification goes out.
The window is rolling, not a fixed bucket. A 15-minute window at 10:04 means 09:49–10:04; a minute later it means 09:50–10:05. Nothing waits for a clock boundary.
It fires on the crossing, not on the breach
This is the single most important thing to know about alerts, and it is what keeps a channel worth reading.
An alert notifies once, at the moment it goes from healthy to breaching. While it stays breached, nothing more is sent — a latency problem that lasts an hour is one message, not sixty. When the value comes back under your threshold the alert quietly returns to OK; no "recovered" message is sent either. If it breaches again later, that is a new crossing and you hear about it again.
This is deliberate, and it is what LangSmith, Sentry and every alerting tool people actually keep enabled do. An alert that repeats every minute while a problem is ongoing trains you to mute the channel — and then you miss the next one.
The three states
| State | What it means |
|---|---|
| OK | The number was computed and it is inside your threshold. |
| Alerting | The number is breaching. It notified on the crossing and will not notify again until it recovers first. |
| No data | There was nothing to measure, or not enough. The alert cannot fire from this state — see Not enough data. |
| Paused | You turned the alert off. It is not evaluated at all. |
What alerts do not do
Worth knowing up front, so you don't go looking:
- No grouping. An alert produces one number for the whole window, not one per agent or per tool. To watch three agents separately, make three alerts with a filter each — which also lets you give each one its own threshold, which you usually want anyway.
- No "recovered" notification. Only the crossing into breach is announced.
- No anomaly detection. An alert compares against the fixed number you typed, not against last week. If you want "this is unusual" rather than "this is over the line", that is what Signals are for.
Next
- Create an alert — the form, field by field
- Measures and aggregations — the full catalog of what you can watch
- Filters — narrowing what counts
- Notifications — Slack, webhooks, and the message
- Monitoring alerts — the list, the chart, the history
AI Score
Seven generation-quality scores grade every LLM node output — grounding, rule adherence, contradiction, trajectory, factual retention, echo, and refusal. What each means, its range, and how to improve it.
Overview
Alerts watch one number about your runs or spans — latency, cost, error rate, tokens, tool calls — and notify Slack or a webhook the moment it crosses a threshold you set.