Monitor an endpoint

Follow an endpoint’s load with charts for GPU, CPU, and memory. For vLLM, SGLang, and vLLM-Omni endpoints, charts also show the serving speed. Use them to decide when to change the workers or the machine.

Before you begin

• Permission to read model endpoints in the project. A Member needs a policy that grants read permission for the Inference service.
• An endpoint that is serving. The Monitoring tab says “Metrics are not available for the current service status.” while the endpoint is Stopped, Terminating, Error, or Idle.

Open the charts

1. Open the endpoint and select the Monitoring tab.
2. Choose Hardware for GPU, CPU, and memory use, which every endpoint has. Choose Service for serving speed: it has charts for vLLM, SGLang, and vLLM-Omni endpoints, and for a Custom Container that runs a vLLM or SGLang image.
3. Choose All Workers, which is available when there is more than one worker, or choose a single worker.
4. Choose a time range of 5m, 15m, 30m, 1h, 1d, or 7d. Refresh for the latest data.

Read the hardware charts

On a GPU endpoint, Hardware shows readings for AVG GPU UTIL, AVG VRAM, AVG GPU MEMORY, AVG TEMP, MAX TEMP, TOTAL POWER, AVG CPU, and AVG MEMORY. For one worker, the readings are GPU UTIL, VRAM, GPU MEMORY, AVG TEMP, MAX TEMP, POWER, CPU, and MEMORY. Charts below them follow GPU utilization, GPU memory used, GPU temperature, GPU power draw, CPU utilization, and CPU memory used. A CPU endpoint shows only the CPU and memory readings and charts.

With more than one worker on a GPU endpoint, choose All Workers and Hardware, then select Compare Replicas. Real Time GPU Utilization by Worker compares GPU utilization across the workers.

Read the serving charts

For vLLM and SGLang endpoints, Service shows readings and charts for Request Rate, TTFT p95, ITL p95, Token Throughput, TTFT SLO Compliance, Prefix Cache Hit Rate, Preemptions / sec, and Achieved TFLOP/s per GPU. TTFT p95 is the time to the first token, and ITL p95 is the gap between tokens. Use them to tell whether callers wait too long for the first token, and add workers when they do. A vLLM-Omni endpoint shows its own set of serving charts.

Set an alert

Open the Settings tab. The Alerts block manages the endpoint’s alerts in Alert Management. An alert sends you an email when a monitored metric crosses its limit, so you can act before performance or availability suffers. See Attach an alert to a resource.

Check the charts

To follow what a worker reports, see View logs and events.