An order sync fails on Friday evening. The alert goes to a shared channel that receives forty messages a day, most of them harmless warnings. Nobody opens it until a customer asks on Monday why their order never shipped.
The automation did send an alert. The problem was that nobody could tell it mattered more than the others.
Decide what deserves an alert
Start from business impact, not from every error the system can produce. Ask for each failure type: if nobody acts within a day, what happens?
- Act now: customers, payments, or deadlines are affected. Notify a named person directly.
- Act today: work is delayed but recoverable. Add it to a queue someone checks on a schedule.
- Review weekly: retries that succeeded, minor slowdowns, or patterns to watch. Put these in a summary, not a notification.
If an alert never leads to action, change its level or remove it. A noisy channel trains people to ignore the important message.
Write alerts a person can act on
A useful alert answers four questions without opening a log:
- What failed: "Order sync to the warehouse system," not "Job 7 error."
- What is affected: how many records, which customers or dates, and whether work is stuck.
- What already happened: retries attempted, records skipped, anything partly written.
- What to do next: a link to the exception queue, the runbook, or the record itself.
Keep sensitive details out of the alert text. Link to the system where access is controlled.
Send each alert to a named owner
Every alert type should have an owner and a backup. A channel is fine for visibility, but someone specific needs to be responsible for acknowledging it. Agree on what happens outside working hours, even if the answer is "the next morning" for most failures.
Group repeats and confirm recovery
If the same failure happens two hundred times, send one alert with a count, not two hundred messages. When the issue clears, send a recovery note so people know whether they still need to act.
Review the alerts every month
Look back at what fired. Which alerts led to action? Which were ignored, duplicated, or arrived too late? Adjust the levels and wording, and remove alerts no one needs.
Alerts are the start of the response, not the end. For the work that follows, see our automation exception queue guide. To review the alerts on an existing automation, email Storm Nova Labs.
