---
title: "Alert metrics and conditions"
description: "The metrics each Alert Management service offers, how severity, operator, and threshold form a condition, and thresholds that work well."
content_type: "concept"
product: "alerts"
platform: "TIR"
url: "/tir/alerts/concepts/alert-metrics-and-conditions/"
---

# Alert metrics and conditions

An alert compares one metric of a resource with a threshold.

## The condition

A condition has four parts you choose in [Create an alert](/tir/alerts/how-to/create-an-alert/): the
metric (**Alert trigger type**), the **Operator**, the **Threshold value**, and the **Severity**.

- **Operator** is **Greater Than**, **Less Than**, **Greater Than or Equal To**, **Less Than or
  Equal To**, **Equal To**, or **Not Equal To**.
- **Threshold value** is a number in the metric's unit. For a percentage metric it is from 0 to 100.
- **Severity** is **Critical**, **Warning**, or **Info**. It labels the alert in the list, in
  notifications, and in the webhook request, and the alert fires the same way whatever its severity.

The alert's **Status** reads **Firing** as soon as the condition is met. TIR sends the firing
notification when the condition has stayed true for five minutes. The status returns to **OK** when
the value is back within the threshold, and TIR then sends a resolved notification. When a resource
stops reporting data, for example because it is stopped, its alert counts as **OK** and stays silent.

**XID Error** has a fixed condition, **Not Equal To** 0, so it fires when the GPU reports an XID
error. **Nodes Down** fires whenever more than 0 nodes are down.

## Metrics by service

The **Service for alert** you choose decides the metrics. When you attach the alert, TIR refuses a
metric outside its service's list.

| Metric                  | Unit    | Instance | VM  | Inference | Training Cluster |
| ----------------------- | ------- | :------: | :-: | :-------: | :--------------: |
| **CPU Utilization**     | %       |   yes    | yes |    yes    |       yes        |
| **Memory Utilization**  | %       |   yes    | yes |    yes    |       yes        |
| **Disk Utilization**    | %       |   yes    |     |           |                  |
| **Ephemeral Storage**   | %       |   yes    |     |           |                  |
| **GPU Temperature**     | C       |   yes    | yes |    yes    |       yes        |
| **GPU Power**           | W       |   yes    | yes |    yes    |                  |
| **GPU Memory Used**     | %       |          |     |           |       yes        |
| **XID Error**           |         |          |     |           |       yes        |
| **Network Bandwidth**   | Bytes/s |          | yes |           |                  |
| **Network Packet Rate** | pkt/s   |          | yes |           |                  |
| **Storage IOPS**        | IOPS    |          | yes |           |                  |
| **Storage Throughput**  | Bytes/s |          | yes |           |                  |
| **Nodes Down**          | count   |          |     |           |       yes        |
| **Pending Jobs**        | count   |          |     |           |       yes        |
| **Failed Jobs**         | count   |          |     |           |       yes        |
| **Node-failed Jobs**    | count   |          |     |           |       yes        |

GPU metrics attach only to resources on a GPU plan. The network and storage metrics attach only to a
VM.

## Load Balancer metrics

A Load Balancer has its own metrics. A rate or a share is measured over the last 5 minutes.

| Metric                        | Unit  | What it measures                                                                                                                    |
| ----------------------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Error Rate (5xx)**          | %     | The share of requests answered with a 5xx status.                                                                                   |
| **Avg Response Time**         | ms    | The mean time to answer a request.                                                                                                  |
| **Healthy Backends %**        | %     | The share of backend targets passing health checks, counted across all backend groups.                                              |
| **Requests / sec**            | req/s | The request rate.                                                                                                                   |
| **Active Connections**        | none  | The number of open client connections.                                                                                              |
| **Rate-Limited (429) / sec**  | req/s | The requests the load balancer's own rate limit turned away.                                                                        |
| **Certificate Expiry (days)** | days  | The days left on the certificate that expires first.                                                                                |
| **Capacity Utilization %**    | %     | The higher of two shares: the request rate against the plan's request limit, and the open connections against its connection limit. |

A TCP load balancer accepts alerts on two of these metrics: **Healthy Backends %** and **Active
Connections**. When you create the alert from the **Alerts** tab of a TCP load balancer, the form
lists only these two.

What each metric helps you find:

- **CPU Utilization**: a CPU running at its limit.
- **Memory Utilization**: memory running low, before the resource runs out.
- **Disk Utilization**: storage running out on volumes.
- **Ephemeral Storage**: temporary container storage filling up.
- **GPU Temperature**: heat that can slow or damage the GPU.
- **GPU Power**: abnormal power use.
- **XID Error**: GPU errors.

## Thresholds that work well

- For capacity planning, set **Memory Utilization** and **Disk Utilization** alerts at 75 to 80
  percent. You get an early warning and time to scale or clean up.
- To watch GPU health in long training runs, set **GPU Temperature** and **GPU Power** alerts to
  catch overheating or abnormal power use.
- For incident response, match your monitoring integrations to your on-call rotation so the right
  people are notified at once.