Monitoring alerts
Read the alerts list, the value-over-time chart with its breach band, how often an alert has triggered, and the full history of every fire.
Alerts aren't only for notifying. Once one exists, the product keeps a record of the number over time — which is often more useful than the notification, because it shows you whether a threshold was right.
The list
Assess → Alerts shows every alert, ordered for noticing rather than filing: Alerting first, then OK, then No data, then paused. Each row carries its state, what it watches in one line, its last value, how many times it triggered in the last 30 days, and when it last fired.
A Not delivered badge means the most recent notification didn't arrive — see When delivery fails.
The row menu has Pause / Resume, Edit and Delete.
The alert page
Open an alert for the detail:
Two stats at the top — how many times it triggered in the last 30 days, and when it last triggered.
The value over time. The measure plotted across 6h, 24h, 7d or 30d. Two things are drawn on it:
- Your threshold, as a line — so you can see how much headroom you normally have.
- The breach band, shading everything past the threshold in red. If your alert is "latency above 5 s", everything above 5 s on the chart has a red background, and the shape of the problem is visible at a glance.
- Fire markers where the alert actually crossed.
Times triggered, as bars per day over the last 30 days. A wall of bars means the threshold is too tight; a flat empty chart on an alert you expected to fire means it is too loose, or the filter matches nothing.
History, a table of every fire: when, the value, the threshold at the time, the sample size, the window, and whether delivery succeeded.
The chart is the fastest way to set a threshold well. Create the alert with a deliberately extreme threshold so it never fires, leave it a day, then read the chart to see where your normal actually sits — and set the real threshold above the noise.
Not enough data
An alert in No data means one of:
- Nothing matched. The window was empty, or your filter matched no rows. On a short window with light traffic this is normal at night.
- The measure had no values. There were rows, but none recorded that number — an alert on cost where no cost was reported, for example.
- Below the sample floor. Fewer rows than the minimum needed to trust the number. For
rateaggregations that floor is at least 5.
An alert cannot fire from No data. That is the point: a number computed from one row is not a number worth waking someone for.
If an alert sits in No data when you expect traffic, check in this order: the filter (a typo in an agent name matches nothing), the scope (a run field on a span alert), then the window (too short for your volume).
Retention
Evaluations — the points on the chart — are kept for 30 days, which is why 30 days is the longest range. The record of each fire is kept as long as the alert is.
Deleting an alert deletes its history with it. Pause instead if you might want it back.
Tuning an alert that isn't working
It fires too often. Raise the threshold, or lengthen the window so brief spikes average out, or raise the minimum sample so quiet periods can't trip it. Check the chart first — if the line is normally close to your threshold, the threshold is the problem, not the noise.
It never fires when it should have. Check the state. If it's No data, the filter or window is wrong. If it's OK, read the chart: the value probably didn't reach your threshold, and an average may be hiding a tail a p95 would catch.
It fires once and then goes quiet during a long incident. Working as intended — an alert notifies on the crossing, not for the duration. See When Trodo sends.
Next
- Create an alert
- Notifications
- Signals — the failures Trodo finds without being asked