Cluster statuses and actions

A TIR Slurm Cluster is always in one of nine statuses, and the status decides which tabs open and which actions the Actions menu offers. When an action you expect is missing, the status is the first thing to check.

The nine statuses

A cluster’s status is one of Creating, Initializing Worker Nodes, Health Check Running, Running, Failed, Terminating, Terminated, Deleting, or Deleted.

A cluster created in the portal moves from Creating to Initializing Worker Nodes to Running, and skips Health Check Running, which appears only when health checks were requested at creation.

Diagram as text

flowchart LR
    C[Creating] --> I[Initializing Worker Nodes]
    I --> R[Running]
    I -.-> H[Health Check Running]
    H -.-> R
    C --> F[Failed]
    C --> T
    I --> T
    F --> T
    R --> T[Terminating]
    T --> TD[Terminated]
    TD --> D[Deleting]
    D --> DD[Deleted]

In words: a new cluster is Creating while it is set up. It then reaches Initializing Worker Nodes and then Running. A cluster whose creation fails ends in Failed. From any of these, Terminate Cluster moves a cluster through Terminating to Terminated, and Delete moves it through Deleting to Deleted. The dotted path is the one portal clusters skip.

Status	What it means
Creating	The cluster is being set up. This usually takes a few minutes.
Initializing Worker Nodes	The controller and login node are up and the workers are still joining. You can already use the cluster.
Health Check Running	Checks requested at creation are running on the workers. You can already use the cluster.
Running	The cluster is live and accepting jobs.
Failed	Creation failed after you submitted the form. Its reserved nodes are released, and its security group and storage selections are cleared.
Terminating	Termination is in progress.
Terminated	The cluster is terminated. The record stays in the list so you can clone or delete it.
Deleting	Deletion is in progress.
Deleted	The cluster is deleted and leaves the list.

A Failed cluster stays in the list until you delete it, and its compute is free. Use Clone Cluster to start a new create form from it. See Clone a cluster. If a cluster sits in Creating for much longer than a few minutes, see Troubleshooting TIR Slurm Cluster.

The cluster list refreshes itself while any cluster is Creating, Initializing Worker Nodes, Health Check Running, or Deleting. To reload the details page, select refresh in its header.

Which tabs open in each status

The Details tab is always available. The other tabs open as the cluster comes up:

Status	Tabs that open
Creating	Details only
Initializing Worker Nodes	Details and Nodes
Health Check Running	Details, Nodes, Jobs, Monitoring, and Logs
Running	Every tab, including Volumes and Network & Security

You can connect while the cluster is in Initializing Worker Nodes, Health Check Running, or Running. Before that, Connect and the closed tabs give a reason: “Available once the login node is up” while Creating, and “Available when the cluster is running” otherwise.

Which actions each status offers

You open the Actions menu from a cluster’s row in the list or from the Actions button on its details page. Both offer the same actions, with the destructive ones last, and show only the actions your permissions allow.

Status	Actions offered
Creating, Initializing Worker Nodes, Health Check Running	Reconfigure Cluster, Terminate Cluster, Delete Cluster
Running	Each action in the next table, when its condition is met
Failed	Clone Cluster, Terminate Cluster, Delete Cluster
Terminating, Terminated	Update SSH Keys, Clone Cluster, Terminate Cluster, Delete Cluster

Action	When it is offered
Update SSH Keys, Clone Cluster, Terminate Cluster, Delete Cluster	Every status, except where the list above says otherwise
Update Image, Slurm Configuration, Restart All Workers, Restart Cluster, Restart Login Service	Only while the cluster is Running
Scale Cluster	While Running, on an hourly plan or a TIR Private Cluster. It is hidden on a committed plan.
Convert to Committed	On an hourly plan outside a TIR Private Cluster, in any status except Failed, Terminated, or Terminating
Reconfigure Cluster	While Running or being set up

The details header also has Restart Workers, a shortcut for Restart All Workers, available only while the cluster is Running. Reboot Node is per node and lives on the Nodes tab. See Update a cluster, Scale a cluster, Clone a cluster, Convert a cluster to committed, and Terminate or delete a cluster.

Find out why an action is missing

Check three things in this order:

1. The cluster’s status. Most hidden actions trace to it, because most actions need Running.
2. The plan. A missing Scale Cluster on a Running cluster means a committed plan. A missing Convert to Committed means the cluster is already committed, is on a TIR Private Cluster, or is in one of the statuses above.
3. Your permission. Permissions follow the Read, Update, Create, and Delete grants on Training Clusters. Read covers viewing and Connect. Update covers every action except Clone and Delete, plus partitions, volumes, IPs, security groups, and node reboot. You need Create for Clone Cluster, and Delete for Delete Cluster. If the whole page is read-only, your role needs Update permission, which a project administrator can add by changing your role.

A confirm button in a dialog becomes available once the dialog is complete:

• you have made a selection that differs from the current one (Update SSH Keys, Update Image)
• every validation error the dialog listed is fixed (Slurm Configuration)
• the new total is above the current node count and within the free inventory (scale up)
• at least one node is selected and at least one is left unselected (scale down)
• the acknowledgement is ticked (Convert to Committed, delete a partition, delete a cluster)
• any earlier submission from the dialog has finished

What each action does to your jobs

Actions run straight away, whatever is in the Slurm queue. Restart All Workers, Restart Cluster, scale down, a node reboot, and terminate or delete all end the jobs in flight. Update Image is the one action that drains nodes one at a time, as part of its rolling restart.

Action	Do running jobs end?
Update SSH Keys	No.
Convert to Committed	No. Only the billing changes.
Slurm Configuration	No. Running jobs continue, and queued jobs use the new settings.
Reconfigure Cluster	No.
Scale up	No. New nodes join and running jobs continue.
Mount or unmount a volume	Yes. The cluster’s worker nodes and login node restart.
Update Image	Gracefully. Nodes are drained one at a time.
Restart Login Service	Only shells on the login node. Slurm jobs keep running.
Restart All Workers	Yes, every running job.
Restart Cluster	Yes, every running job.
Scale down	Yes, on the removed nodes, and the remaining workers restart too.
Reboot Node	Yes, the jobs on that node.
Terminate Cluster, Delete Cluster	Yes, everything.

Before any action that ends jobs, open the Jobs tab or run squeue. Let jobs finish, or drain the node first with scontrol update nodename=<node> state=drain reason="<why>", and make sure your jobs checkpoint to a PFS or SFS mount. If a change fails to apply, TIR refuses the request with an error and the cluster’s settings go back to what they were.

For the errors a refused action returns, see Troubleshooting TIR Slurm Cluster. For the parts of the details page, see The cluster details page.