Monitor a private cluster

Check a private cluster’s nodes, see what each project runs on it, and review who changed what. All of it is on the cluster’s own page, set up for you.

Before you begin

You need a private cluster with nodes allocated to a project. See Create a private cluster and Allocate nodes to a project.

Check a node’s health and usage

1. In Manage Private Cluster, open the cluster.
2. Select the Node Monitoring tab, and choose a time window: 1 hour, 24 hours, 7 days, or 30 days.

Each node has its own panel with a health badge of HEALTHY or UNHEALTHY, its status, and charts of Memory, Uptime, Power, and the node’s GPU utilization, which the panel labels CPU. A node that shows UNHEALTHY while a workload will not start on it is worth replacing; see A workload will not start on an allocated node.

Check the cluster’s configuration

The Cluster Details tab shows the cluster’s name, creation time, and status, and its total, available, and allocated node counts. It also shows the plan name, whether the cluster is billed Hourly or Committed, and the GPU, GPU memory, CPU, and RAM of its nodes.

See what a project is running

Open a project’s row in Project Allocations to see its Deployed Services, each with the GPU cards it uses and its status. To check one node instead, open it from Cluster Nodes: its own Deployed Services and Monitoring tabs cover that node alone. See Launch services on a private cluster.

Review the cluster’s audit log

Select the Audit Logs tab. It lists, newest first, the timestamp, project name, event, number of nodes the event moved, and the client IP it came from. Events recorded include creating and deleting the cluster, changing its configuration, allocating, deallocating, or moving nodes, and creating, updating, or deleting a node-affinity group. An interval filter narrows it to the current month and similar ranges.

You can now tell how busy each node is, what runs where, and who changed the cluster. For help with a node that will not start, or an allocation that has not shown up, see Troubleshooting TIR Private Cluster.