Create an alert
Walk through the alert form: scope, measure, aggregation, filters, threshold, window, message and destination.
Alerts live under Assess → Alerts in the sidebar. New alert opens a pop-up with three ways in:
- A template. Search or pick one: Hard failures, Tool failures, Failed runs, Failures of one kind (timeouts, rate limits, auth, server errors and the other error categories), Failed runs of one kind, Error rate, Slow runs, Slow tool and Cost spike. The form opens filled in; change anything before you save.
- Create with Lucid. Describe it in plain words and Lucid proposes the alert as an editable card. See Lucid.
- Start from scratch. An empty form, walked through below.
You can also start an alert from a row on the Failures tab, already filtered to that failure, or from Create alert on an eval's page.
Nothing is saved until you hit Save, and an alert starts evaluating as soon as it exists.
1. Name it after the problem
The name is what lands in Slack at two in the morning, so make it the thing you'd want to read then. "Checkout p95 latency" beats "Alert 3".
2. Choose the scope: run or span
| Scope | What one row is | Use it for |
|---|---|---|
| Run | One whole agent execution, end to end. | Total latency, cost per run, overall error rate, users affected. |
| Span | One step inside a run: an LLM call, a tool call, a retrieval. | A specific step: this tool is slow, this model is expensive. |
A third scope, Eval, watches an eval's results: failures, pass rate, scores. It is easiest to start from Create alert on the eval's page; see Alerts on evals.
The scope decides which measures and which filter fields you get. Switching it resets the measure, because a span has no "tool calls" count and a run has no model.
3. Pick the measure and aggregation
The measure is the number: duration, cost, error rate, tokens. The aggregation is how many rows become one: count, rate, avg, sum, min, max, or a percentile (p50 through p99).
Only the aggregations that make sense for a measure are offered. A count of runs has no average; an error rate is always a rate. See Measures and aggregations for the whole catalog.
4. Narrow it with filters (optional)
Without a filter the alert watches every run or span in your team. Add rules to watch a slice: one agent, one tool, one customer, one model.
Rules combine with AND or OR inside a group, and you can add several groups. Filters can also reach into the custom metadata you attach to runs and the attributes you attach to spans. See Filters.
5. Set the condition
The condition reads as a sentence:
Trigger when the value is above
8000ms over the last 15 minutes.
Four operators: above (>), at or above (≥), below (<), at or below (≤). Threshold is a plain number in the measure's own units: milliseconds for duration, dollars for cost, percent for a rate.
The window is any length from 1 minute to 30 days. Common choices are 5, 15 or 30 minutes, an hour, 6 hours, or 24 hours.
Choosing a window
The window is a trade between noticing fast and not jumping at noise.
| Window | Good for | Watch out for |
|---|---|---|
| 5 to 15 min | Outages, latency spikes, error bursts. You want to know now. | On light traffic a short window is often empty or tiny, so the alert sits in No data. |
| 1 to 6 hours | Degradation and drift, costs creeping, quality sliding. | Slow to notice a hard outage. |
| 24 hours+ | Budgets and daily volumes. | A single bad hour barely moves a 24-hour average. |
A good rule: pick the shortest window that still contains a decent number of runs on a normal day.
6. Write the message (optional)
Leave it blank and Trodo composes a sensible default. Write your own and you can drop in variables ({{name}}, {{value}}, {{threshold}}, {{window}}, {{count}}, {{link}} and more) by clicking them below the box. See Notifications.
7. Send it somewhere
Choose Slack, Webhook, or both:
- Slack: pick a channel from your team's connected workspace. Connect Slack first in your integration settings if the picker is empty.
- Webhook: an HTTPS URL that receives a JSON
POSTwhen the alert fires.
A worked example
"Checkout charge step is failing"
| Field | Value |
|---|---|
| Scope | Span |
| Measure / aggregation | Error rate · rate |
| Filter | span.tool_name is charge_card |
| Condition | above 10 % over the last 15 minutes |
| Send to | #eng-oncall |
Every minute Trodo counts the charge_card spans in the last 15 minutes, works out what share of them errored, and, if that's over 10% with at least 5 spans to judge from, posts to #eng-oncall once. It stays quiet while the problem persists, and is ready to tell you again after it clears.
Editing and pausing
Editing an alert resets its state: the old state was about a different condition, so the alert starts fresh at No data and re-evaluates within a minute. That also means changing a threshold on a currently-alerting alert lets it fire again on the next crossing.
Pause an alert from the row menu on the list. A paused alert isn't evaluated at all: no records, no notifications. It resumes cleanly when you turn it back on.
Next
Overview
Alerts watch one number about your runs or spans (latency, cost, error rate, tokens, tool calls) and notify Slack or a webhook the moment it crosses a threshold you set.
Measures & aggregations
The full catalog of what an alert can watch on runs and spans (duration, cost, tokens, error rate, tool calls, users) and which aggregations apply to each.