--- title: "Cluster statuses and actions" description: "The nine statuses of a TIR Slurm Cluster, which tabs and actions each one allows, why an action can be missing, and what each action does to your jobs." content_type: "concept" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/" --- # Cluster statuses and actions A TIR Slurm Cluster is always in one of nine statuses, and the status decides which tabs open and which actions the **Actions** menu offers. When an action you expect is missing, the status is the first thing to check. ## The nine statuses A cluster's status is one of **Creating**, **Initializing Worker Nodes**, **Health Check Running**, **Running**, **Failed**, **Terminating**, **Terminated**, **Deleting**, or **Deleted**. A cluster created in the portal moves from **Creating** to **Initializing Worker Nodes** to **Running**, and skips **Health Check Running**, which appears only when health checks were requested at creation. ```mermaid flowchart LR C[Creating] --> I[Initializing Worker Nodes] I --> R[Running] I -.-> H[Health Check Running] H -.-> R C --> F[Failed] C --> T I --> T F --> T R --> T[Terminating] T --> TD[Terminated] TD --> D[Deleting] D --> DD[Deleted] ``` In words: a new cluster is **Creating** while it is set up. It then reaches **Initializing Worker Nodes** and then **Running**. A cluster whose creation fails ends in **Failed**. From any of these, **Terminate Cluster** moves a cluster through **Terminating** to **Terminated**, and **Delete** moves it through **Deleting** to **Deleted**. The dotted path is the one portal clusters skip. | Status | What it means | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | **Creating** | The cluster is being set up. This usually takes a few minutes. | | **Initializing Worker Nodes** | The controller and login node are up and the workers are still joining. You can already use the cluster. | | **Health Check Running** | Checks requested at creation are running on the workers. You can already use the cluster. | | **Running** | The cluster is live and accepting jobs. | | **Failed** | Creation failed after you submitted the form. Its reserved nodes are released, and its security group and storage selections are cleared. | | **Terminating** | Termination is in progress. | | **Terminated** | The cluster is terminated. The record stays in the list so you can clone or delete it. | | **Deleting** | Deletion is in progress. | | **Deleted** | The cluster is deleted and leaves the list. | A **Failed** cluster stays in the list until you delete it, and its compute is free. Use **Clone Cluster** to start a new create form from it. See [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/). If a cluster sits in **Creating** for much longer than a few minutes, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). The cluster list refreshes itself while any cluster is **Creating**, **Initializing Worker Nodes**, **Health Check Running**, or **Deleting**. To reload the details page, select refresh in its header. ## Which tabs open in each status The **Details** tab is always available. The other tabs open as the cluster comes up: | Status | Tabs that open | | ----------------------------- | -------------------------------------------------------------- | | **Creating** | **Details** only | | **Initializing Worker Nodes** | **Details** and **Nodes** | | **Health Check Running** | **Details**, **Nodes**, **Jobs**, **Monitoring**, and **Logs** | | **Running** | Every tab, including **Volumes** and **Network & Security** | You can connect while the cluster is in **Initializing Worker Nodes**, **Health Check Running**, or **Running**. Before that, **Connect** and the closed tabs give a reason: "Available once the login node is up" while **Creating**, and "Available when the cluster is running" otherwise. ## Which actions each status offers You open the **Actions** menu from a cluster's row in the list or from the **Actions** button on its details page. Both offer the same actions, with the destructive ones last, and show only the actions your permissions allow. | Status | Actions offered | | --------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | **Creating**, **Initializing Worker Nodes**, **Health Check Running** | **Reconfigure Cluster**, **Terminate Cluster**, **Delete Cluster** | | **Running** | Each action in the next table, when its condition is met | | **Failed** | **Clone Cluster**, **Terminate Cluster**, **Delete Cluster** | | **Terminating**, **Terminated** | **Update SSH Keys**, **Clone Cluster**, **Terminate Cluster**, **Delete Cluster** | | Action | When it is offered | | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | | **Update SSH Keys**, **Clone Cluster**, **Terminate Cluster**, **Delete Cluster** | Every status, except where the list above says otherwise | | **Update Image**, **Slurm Configuration**, **Restart All Workers**, **Restart Cluster**, **Restart Login Service** | Only while the cluster is **Running** | | **Scale Cluster** | While **Running**, on an hourly plan or a TIR Private Cluster. It is hidden on a committed plan. | | **Convert to Committed** | On an hourly plan outside a TIR Private Cluster, in any status except **Failed**, **Terminated**, or **Terminating** | | **Reconfigure Cluster** | While **Running** or being set up | The details header also has **Restart Workers**, a shortcut for **Restart All Workers**, available only while the cluster is **Running**. **Reboot Node** is per node and lives on the **Nodes** tab. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/), [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/), [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/), [Convert a cluster to committed](/tir/tir-slurm-cluster/how-to/convert-a-cluster-to-committed/), and [Terminate or delete a cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/). ## Find out why an action is missing Check three things in this order: 1. **The cluster's status.** Most hidden actions trace to it, because most actions need **Running**. 2. **The plan.** A missing **Scale Cluster** on a **Running** cluster means a committed plan. A missing **Convert to Committed** means the cluster is already committed, is on a TIR Private Cluster, or is in one of the statuses above. 3. **Your permission.** Permissions follow the Read, Update, Create, and Delete grants on Training Clusters. Read covers viewing and **Connect**. Update covers every action except Clone and Delete, plus partitions, volumes, IPs, security groups, and node reboot. You need Create for **Clone Cluster**, and Delete for **Delete Cluster**. If the whole page is read-only, your role needs Update permission, which a project administrator can add by changing your role. A confirm button in a dialog becomes available once the dialog is complete: - you have made a selection that differs from the current one (**Update SSH Keys**, **Update Image**) - every validation error the dialog listed is fixed (**Slurm Configuration**) - the new total is above the current node count and within the free inventory (scale up) - at least one node is selected and at least one is left unselected (scale down) - the acknowledgement is ticked (**Convert to Committed**, delete a partition, delete a cluster) - any earlier submission from the dialog has finished ## What each action does to your jobs Actions run straight away, whatever is in the Slurm queue. **Restart All Workers**, **Restart Cluster**, scale down, a node reboot, and terminate or delete all end the jobs in flight. **Update Image** is the one action that drains nodes one at a time, as part of its rolling restart. | Action | Do running jobs end? | | ----------------------------------------- | ----------------------------------------------------------------- | | **Update SSH Keys** | No. | | **Convert to Committed** | No. Only the billing changes. | | **Slurm Configuration** | No. Running jobs continue, and queued jobs use the new settings. | | **Reconfigure Cluster** | No. | | Scale up | No. New nodes join and running jobs continue. | | Mount or unmount a volume | Yes. The cluster's worker nodes and login node restart. | | **Update Image** | Gracefully. Nodes are drained one at a time. | | **Restart Login Service** | Only shells on the login node. Slurm jobs keep running. | | **Restart All Workers** | Yes, every running job. | | **Restart Cluster** | Yes, every running job. | | Scale down | Yes, on the removed nodes, and the remaining workers restart too. | | **Reboot Node** | Yes, the jobs on that node. | | **Terminate Cluster**, **Delete Cluster** | Yes, everything. | Before any action that ends jobs, open the **Jobs** tab or run `squeue`. Let jobs finish, or drain the node first with `scontrol update nodename= state=drain reason=""`, and make sure your jobs checkpoint to a PFS or SFS mount. If a change fails to apply, TIR refuses the request with an error and the cluster's settings go back to what they were. For the errors a refused action returns, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). For the parts of the details page, see [The cluster details page](/tir/tir-slurm-cluster/concepts/the-cluster-details-page/).