Alerts

Overview

An alert watches one number about your runs, spans or an eval's results, tells Slack or a webhook when it crosses your limit, and starts the agents that act on it.

An alert watches one number about your agents and tells you when it crosses a line you drew: tool failures above 5 in an hour, p95 latency over 8 seconds, an eval's pass rate under 90%. Trodo recomputes the number every minute over a rolling window. The moment it crosses, the alert posts to Slack or a webhook and starts any agents that listen to it.

In the loop, alerts sit between what is measured and what gets done: evals score your traces, alerts decide when there is enough of something to act on, and the agents they start (often Lucid) investigate and file issues.

Alerts live under Assess → Alerts.

Assess → Alerts: one row per alert with its state dot, what it watches, its condition and how many agents it starts, how often it fired in the last 30 days, and when it last fired; New alert at the top right.

What an alert is made of

It reads as one sentence: "Spans with status error, count above 5, over the last hour."

PartWhat it is
ScopeWhat one row is: a Run (a whole agent execution), a Span (one step inside it) or an Eval (one eval's results).
MeasureThe number: duration, cost, error rate, tokens, users, an eval's failures or pass rate. Each comes with the aggregations that fit it: a count, a rate, an average, a p95. See Measures & filters.
FilterWhich rows count. Without one, the alert watches every run or span of the team (or every result of its eval); with one, only one agent, one tool, one customer.
ConditionAn operator, a limit and a window: above 8 s over the last 15 minutes. The window is anything from 1 minute to 30 days and is rolling: it always ends now.

How it is evaluated

  1. Every minuteTrodo reads the rows in the window and keeps the ones the filter matches.
  2. One numberThe rows become one value with the aggregation: a count, a rate, an average, a p95.
  3. CompareThe value is compared with the limit, and the alert takes a state.
StateMeans
okThe number was computed and is inside the limit.
alertingThe number is past the limit.
no dataNothing to measure, or too little: the window was empty, no row had a value, or there were fewer rows than the minimum (5 for a rate). An alert can't fire from here.

A paused alert isn't evaluated at all. On the alert's page, Current turns red while it is alerting; in the form, Right now says would trigger, would not trigger or not enough to decide. In the list, the dot only says whether the alert is on or paused.

It fires on the crossing

An alert notifies once, when it goes from ok or no data into alerting. While it stays past the limit nothing more is sent, so a problem that lasts an hour is one message, not sixty. When the value comes back the alert returns to ok without a message. If it crosses again later, that is a new firing.

What a firing does

  • Notifies the Slack channel or webhook you chose. Those are the two destinations. See Notifications & monitoring.
  • Starts agents. Any agent whose trigger is this alert runs, with the alert, its value and the moment it fired as input. The list shows how many agents each alert starts. This is how an alert leads to an issue: an alert never files one itself; an agent with a Lucid step does (see Issues).
  • Uses one Execution. A firing is never held back for billing; only the agents it starts can be. See Plans & billing.
  • Is recorded on the alert's page, delivered or not.

Alerts you don't see here

Some alerts are managed by something else, and the Alerts page leaves them out:

  • Track issues deploys one alert per type it tracks. Change them in Track issues.
  • A self-improving eval has an alert on its own results. Change it in the eval's settings; see Self-improving evals.

Opening a managed alert directly is refused with a pointer to where it is managed. Lucid still sees them when you ask about your alerts.

What alerts don't do

  • No grouping. An alert is one number for the whole window, not one per agent or per tool. To watch three agents separately, make three alerts with a filter each, which also lets each have its own limit.
  • No "recovered" message. Only the crossing into alerting is announced.
  • No anomaly detection. An alert compares against the number you typed, not against last week. Set it from what normal looks like on the alert's chart.

On this page