Alert metrics and conditions

An alert compares one metric of a resource with a threshold.

The condition

A condition has four parts you choose in Create an alert: the metric (Alert trigger type), the Operator, the Threshold value, and the Severity.

• Operator is Greater Than, Less Than, Greater Than or Equal To, Less Than or Equal To, Equal To, or Not Equal To.
• Threshold value is a number in the metric’s unit. For a percentage metric it is from 0 to 100.
• Severity is Critical, Warning, or Info. It labels the alert in the list, in notifications, and in the webhook request, and the alert fires the same way whatever its severity.

The alert’s Status reads Firing as soon as the condition is met. TIR sends the firing notification when the condition has stayed true for five minutes. The status returns to OK when the value is back within the threshold, and TIR then sends a resolved notification. When a resource stops reporting data, for example because it is stopped, its alert counts as OK and stays silent.

XID Error has a fixed condition, Not Equal To 0, so it fires when the GPU reports an XID error. Nodes Down fires whenever more than 0 nodes are down.

Metrics by service

The Service for alert you choose decides the metrics. When you attach the alert, TIR refuses a metric outside its service’s list.

Metric	Unit	Instance	VM	Inference	Training Cluster
CPU Utilization	%	yes	yes	yes	yes
Memory Utilization	%	yes	yes	yes	yes
Disk Utilization	%	yes
Ephemeral Storage	%	yes
GPU Temperature	C	yes	yes	yes	yes
GPU Power	W	yes	yes	yes
GPU Memory Used	%				yes
XID Error					yes
Network Bandwidth	Bytes/s		yes
Network Packet Rate	pkt/s		yes
Storage IOPS	IOPS		yes
Storage Throughput	Bytes/s		yes
Nodes Down	count				yes
Pending Jobs	count				yes
Failed Jobs	count				yes
Node-failed Jobs	count				yes

GPU metrics attach only to resources on a GPU plan. The network and storage metrics attach only to a VM.

Load Balancer metrics

A Load Balancer has its own metrics. A rate or a share is measured over the last 5 minutes.

Metric	Unit	What it measures
Error Rate (5xx)	%	The share of requests answered with a 5xx status.
Avg Response Time	ms	The mean time to answer a request.
Healthy Backends %	%	The share of backend targets passing health checks, counted across all backend groups.
Requests / sec	req/s	The request rate.
Active Connections	none	The number of open client connections.
Rate-Limited (429) / sec	req/s	The requests the load balancer’s own rate limit turned away.
Certificate Expiry (days)	days	The days left on the certificate that expires first.
Capacity Utilization %	%	The higher of two shares: the request rate against the plan’s request limit, and the open connections against its connection limit.

A TCP load balancer accepts alerts on two of these metrics: Healthy Backends % and Active Connections. When you create the alert from the Alerts tab of a TCP load balancer, the form lists only these two.

What each metric helps you find:

• CPU Utilization: a CPU running at its limit.
• Memory Utilization: memory running low, before the resource runs out.
• Disk Utilization: storage running out on volumes.
• Ephemeral Storage: temporary container storage filling up.
• GPU Temperature: heat that can slow or damage the GPU.
• GPU Power: abnormal power use.
• XID Error: GPU errors.

Thresholds that work well

• For capacity planning, set Memory Utilization and Disk Utilization alerts at 75 to 80 percent. You get an early warning and time to scale or clean up.
• To watch GPU health in long training runs, set GPU Temperature and GPU Power alerts to catch overheating or abnormal power use.
• For incident response, match your monitoring integrations to your on-call rotation so the right people are notified at once.