E2E CloudWhat a notification contains
TIR sends a notification when an alert starts firing, and a second one when it is resolved. Both go to every monitoring integration the alert uses, so an alert with an email, a Slack, and a webhook integration notifies all three.
When notifications are sent
Section titled “When notifications are sent”- Firing: the metric has met the condition for five minutes. The status in the resource’s alert table reads Firing from the moment the condition is met, so it can change before the notification arrives.
- Resolved: the value is back within the threshold. The status returns to OK.
- The alert can fire again each time the condition is met again after it resolved.
- An alert fires only while its resource reports data, so an alert on a stopped Instance stays silent.
- An integration sends only when it has a valid destination, such as at least one member for email. Otherwise it fails silently.
- For a Load Balancer alert, the resource in the notification is the load balancer. A webhook
request lists that one load balancer, with
replicaandgpu_uuidempty.
Each member of the integration receives an email at the address of their account. A member who is in several integrations of the alert receives one email. The email names the alert, the resource, the project, the severity, the metric, the operator, the threshold, and the time of the event. It says whether the alert fired or resolved.
The message is titled [FIRING:1] <alert name> <project name> or [RESOLVED:1] ..., where the
number is how many resources are affected. The body gives the alert name, the resource name, the
project name, the severity, the metric, the operator, and the threshold. The message also links to
the TIR portal.
Slack delivery problems show up only when you use Test URL or Test Token in the integration form. A live message can fail silently, for example when the bot has yet to join the channel. See Troubleshooting Slack alerts.
Webhook
Section titled “Webhook”The request is a POST with a JSON body. It carries the alert, the project, and the affected resources. TIR sends it once and waits up to 10 seconds. See Webhook requests for every field.