Creating an Alert in Trodo

Walk through the alert form: scope, measure, aggregation, filters, threshold, window, message and destination.

Alerts live under Assess → Alerts in the sidebar. New alert opens the form. Nothing is saved until you hit Save, and an alert starts evaluating as soon as it exists.

1. Name it after the problem

The name is what lands in Slack at two in the morning, so make it the thing you'd want to read then. "Checkout p95 latency" beats "Alert 3".

2. Choose the scope: run or span

Scope	What one row is	Use it for
Run	One whole agent execution, end to end.	Total latency, cost per run, overall error rate, users affected.
Span	One step inside a run — an LLM call, a tool call, a retrieval.	A specific step: this tool is slow, this model is expensive.

The scope decides which measures and which filter fields you get. Switching it resets the measure, because a span has no "tool calls" count and a run has no model.

When you're not sure: if you'd investigate it by opening a run, scope to runs. If you'd investigate it by opening the waterfall inside a run, scope to spans.

3. Pick the measure and aggregation

The measure is the number — duration, cost, error rate, tokens. The aggregation is how many rows become one: count, rate, avg, sum, min, max, or a percentile (p50 through p99).

Only the aggregations that make sense for a measure are offered. A count of runs has no average; an error rate is always a rate. See Measures and aggregations for the whole catalog.

For latency, prefer p95 over avg. An average hides the tail — the 5% of users having a bad time — behind everyone who had a fine time. p95 is the number your users actually complain about.

4. Narrow it with filters (optional)

Without a filter the alert watches every run or span in your team. Add rules to watch a slice: one agent, one tool, one customer, one model.

Rules combine with AND or OR inside a group, and you can add several groups. Filters can also reach into the custom metadata you attach to runs and the attributes you attach to spans. See Filters.

5. Set the condition

The condition reads as a sentence:

Trigger when the value is above 8000 ms over the last 15 minutes.

Four operators: above (>), at or above (≥), below (<), at or below (≤). Threshold is a plain number in the measure's own units — milliseconds for duration, dollars for cost, percent for a rate.

The window is any length from 1 minute to 30 days. Common choices are 5, 15 or 30 minutes, an hour, 6 hours, or 24 hours.

Choosing a window

The window is a trade between noticing fast and not jumping at noise.

Window	Good for	Watch out for
5–15 min	Outages, latency spikes, error bursts. You want to know now.	On light traffic a short window is often empty or tiny, so the alert sits in No data.
1–6 hours	Degradation and drift — costs creeping, quality sliding.	Slow to notice a hard outage.
24 hours+	Budgets and daily volumes.	A single bad hour barely moves a 24-hour average.

A good rule: pick the shortest window that still contains a decent number of runs on a normal day.

Rates need at least 5 rows. One failing run out of one is a 100% error rate — so an error-rate alert at 20% would fire on any quiet minute containing a single failure. Trodo enforces a floor of 5 rows in the window for rate aggregations, and the form says so. Set a higher floor yourself if your traffic supports it.

6. Write the message (optional)

Leave it blank and Trodo composes a sensible default. Write your own and you can drop in variables — {{name}}, {{value}}, {{threshold}}, {{window}}, {{count}}, {{link}} and more — by clicking them below the box. See Notifications.

7. Send it somewhere

Choose Slack, Webhook, or both:

Slack — pick a channel from your team's connected workspace. Connect Slack first in your integration settings if the picker is empty.
Webhook — an HTTPS URL that receives a JSON POST when the alert fires.

An alert with no destination still evaluates and still shows its state in the product — it just doesn't tell anyone. That's fine while you're tuning a threshold, and a common way to have an alert that quietly does nothing.

A worked example

"Checkout charge step is failing"

Field	Value
Scope	Span
Measure / aggregation	Error rate · `rate`
Filter	`span.tool_name` is `charge_card`
Condition	above `10` % over the last `15 minutes`
Send to	`#eng-oncall`

Every minute Trodo counts the charge_card spans in the last 15 minutes, works out what share of them errored, and — if that's over 10% with at least 5 spans to judge from — posts to #eng-oncall once. It stays quiet while the problem persists, and is ready to tell you again after it clears.

Editing and pausing

Editing an alert resets its state: the old state was about a different condition, so the alert starts fresh at No data and re-evaluates within a minute. That also means changing a threshold on a currently-alerting alert lets it fire again on the next crossing.

Pause an alert from the row menu on the list. A paused alert isn't evaluated at all — no records, no notifications — and resumes cleanly when you turn it back on.

Create an alert