--- title: Action Availability and Blockers sidebar_label: Action availability --- import { Eye, Lock, AlertCircle, Activity, List } from 'react-feather'; # Action Availability and Blockers An action can be missing from the menu, greyed out inside a dialog, or rejected after you confirm. This page is the single reference for why — start here when a control you expected is not there. }, { href: '#cluster-statuses', label: 'Cluster statuses', icon: }, { href: '#what-each-status-allows', label: 'What each status allows', icon: }, { href: '#visibility-rules-by-action', label: 'Visibility by action', icon: }, { href: '#blockers-and-their-messages', label: 'Blockers & messages', icon: }, { href: '#permissions', label: 'Permissions', icon: }, ]} /> --- ## Quick Diagnosis Work down this list — the first match is almost always the answer. | Symptom | Check | |---------|-------| | The action is **not in the menu at all** | The cluster status. Most actions need **Running** — see [what each status allows](#what-each-status-allows) | | **Scale Cluster** is missing on a Running cluster | The cluster is on a **committed** plan. Scaling is not supported for committed plans | | **Convert to Committed** is missing | The cluster is already committed, is a **CPU cluster**, or runs on a **private cluster** plan | | **Slurm Configuration** is missing | The cluster is not **Running** | | **Update Image** is missing | The cluster is not **Running** | | No actions at all, or the page is read-only | You do not have **update** permission on Training Clusters in this project | | The dialog opens but the confirm button is grey | Something in the dialog is incomplete — see [disabled-confirm reasons](#why-a-confirm-button-is-disabled) | | You confirmed and got an error | See [blockers and their messages](#blockers-and-their-messages) | | A **tab** shows "available when the cluster is running" | Cluster Overview, Partitions, Nodes, Jobs, Monitoring, Volumes and Network & Security need a Running cluster | | The cluster is gone from the list | It was **deleted**. From the moment Delete is issued the cluster is removed from the list and every action on it fails | --- ## Cluster Statuses | Status | Meaning | |--------|---------| | **Creating** | Provisioning is in progress | | **Running** | The cluster is live and accepting jobs | | **Failed** | Creation did not complete and was rolled back | | **Terminating** | Termination in progress | | **Terminated** | Terminated. The record remains so you can clone or delete it | | **Deleting** | Deletion in progress | | **Deleted** | Deleted. No longer listed | The list page **refreshes itself every 5 seconds** while any cluster is **Creating** or **Deleting**, so you can watch provisioning without reloading. The details page does not auto-refresh — use the refresh icon in the header. :::info Creating usually finishes in a few minutes If a cluster sits in **Creating** much longer, try [**Reconfigure Cluster**](/docs/tir/SlurmCluster/manage/actions#reconfigure-cluster) — it is available on a Creating cluster and re-applies the intended state. If that does not help, note the cluster name and contact support. ::: --- ## What Each Status Allows | Status | Actions offered | |--------|-----------------| | **Creating** | Update SSH Keys · Convert to Committed¹ · Reconfigure Cluster · Clone Cluster · Terminate Cluster · Delete | | **Running** | Everything | | **Failed** | Update SSH Keys · Clone Cluster · Terminate Cluster · Delete | | **Terminating** | Update SSH Keys · Clone Cluster · Terminate Cluster · Delete | | **Terminated** | Update SSH Keys · Clone Cluster · Terminate Cluster · Delete | | **Deleting** / **Deleted** | Not listed. Once Delete is issued the cluster leaves the list and no action on it is possible | ¹ Only on hourly, non-private-cluster plans. :::tip Failed clusters are for cloning, not fixing A **Failed** cluster cannot be repaired into a Running one. The productive path is **Clone Cluster** — you get the same configuration in a fresh create form — then **Delete** the failed record. A Failed cluster is not billed for compute. ::: --- ## Visibility Rules by Action | Action | Visible when | Hidden when | |--------|--------------|-------------| | Update SSH Keys | Always | Never | | Update Image | Status is Running | Any other status | | Slurm Configuration | Status is Running | Any other status | | Scale Cluster | Running **and** (hourly plan **or** private cluster) | Committed plan, or not Running | | Convert to Committed | Hourly plan **and** not a private cluster **and** status is not Failed / Terminating / Terminated | Committed plan, CPU cluster, private cluster, or one of those statuses | | Restart All Workers | Running | Any other status (the header button is disabled rather than hidden) | | Restart Cluster | Running | Any other status | | Restart Login Service | Running | Any other status | | Reconfigure Cluster | Running **or** Creating | Any other status | | Clone Cluster | Always | Never | | Terminate Cluster | Always | Never | | Delete | Always | Never | | Add / Edit / Delete Partition | Running; row actions only on **your** partitions | Platform partitions show `Managed` and have no actions | | Node reboot | Nodes tab, and the node is not already rebooting | While a reboot is in flight | | Mount / unmount a volume | Volumes tab, cluster Running | — | | IP and security-group actions | Network & Security tab, cluster Running | The VPC section is not shown in every region | | **Connect** | Always in the list; the details button needs Running | The list icon is disabled on a Failed cluster | --- ## Why a Confirm Button Is Disabled | Dialog | Confirm stays grey while | |--------|-------------------------| | Update SSH Keys | Nothing is selected, or the selection is unchanged from the current keys | | Update Image | No version chosen, or the chosen version is already the current one | | Slurm Configuration | The configuration is still loading or being saved. Validation errors are listed above the buttons and block the save | | Scale up | The total is not above the current node count, no inventory is available, or you asked for more than is free | | Scale down | No node selected, or **every** node selected — at least one must remain | | Convert to Committed | No committed plan is available, or the acknowledgement is unticked | | Delete Partition | The acknowledgement checkbox is unticked | | Delete Training Cluster | The acknowledgement checkbox is unticked | | Any action | A previous submission is still in flight | --- ## Blockers and Their Messages These are checked after you confirm. ### Plan and cluster-type blockers | Message | Cause | Fix | |---------|-------|-----| | *Scale cluster is not supported for committed cluster* | Scaling a committed plan | Size the cluster before committing, or create a second hourly cluster | | *Cluster plan upgrade not allowed for committed clusters* | Asking for plan options on a committed cluster | None — the commitment window has to run out | | *Convert to committed is not supported for committed cluster* | Already committed | — | | *Convert to committed is not supported for CPU cluster* | CPU clusters have no committed plans | — | ### Capacity and quota blockers | Message | Cause | Fix | |---------|-------|-----| | *Inventory not available for the selected configuration* | No free nodes of the right type on the same rack as your existing nodes | Retry later; inventory is refreshed frequently. For guaranteed capacity, use a [Private Cluster](/docs/myaccount/private_cluster/) | | **Inventory Unavailable** in the Scale dialog | Same, surfaced before you submit | As above, or scale to the maximum the dialog offers | | A credit-related message | Not enough credit for the requested nodes or the upfront committed charge | Top up, then retry | | A BOM or limit message | An account-level limit on the requested configuration | Contact support | ### State blockers | Message | Cause | Fix | |---------|-------|-----| | *Slurm configuration can only be updated on a running cluster* | Editing Slurm config on a non-Running cluster | Wait for **Running** | | *Minimum one node required for cluster to remain in running* | Scaling down to zero | Keep at least one node, or terminate instead | | *Invalid request, Cluster does not exist* | The cluster was deleted, or belongs to a different project | Check the project selector | | A VPC-IP-in-progress error | A VPC IP is mid-attach, which blocks terminate and delete | Wait for the attach to settle, then retry | ### Storage blockers | Message | Cause | Fix | |---------|-------|-----| | *Cannot remove storage. At least one storage volume must remain mounted on the cluster.* | Unmounting the last volume | Mount a replacement first, then unmount | | *Cannot unmount this storage as it is used for persistent login. Disable persistent login on the cluster first.* | The volume holds the home directories | Disable [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management) first | ### Network blockers | Message | Cause | Fix | |---------|-------|-----| | *At least one IP (reserved or floating) must be attached.* | Detaching the last IP | Attach a reserved IP first, then detach the other | | A "at least one security group" error | Removing the last security group | Attach a replacement first | ### Partition validation See [Manage partitions](/docs/tir/SlurmCluster/slurm-configuration/partitions#rules-summary) for the full list, including reserved names, the 20-partition limit, and the node-membership rules. --- ## No Action Checks Your Job Queue This is the most important operational caveat on the page. :::danger Nothing stops you restarting a cluster with jobs running No action inspects the Slurm queue before it runs. **Restart All Workers**, **Restart Cluster**, **Scale down** and **Terminate** all proceed with jobs in flight, and those jobs are lost. Only **Update Image** drains nodes as part of its rolling restart. Before any disruptive action: 1. Open the [**Jobs** tab](/docs/tir/SlurmCluster/manage/jobs) or run `squeue`. 2. Let jobs finish, or drain first: `scontrol update nodename= state=drain`. 3. Confirm your jobs checkpoint to a PFS or SFS mount so a loss costs minutes, not days. ::: ### What each action interrupts | Action | Interrupts jobs? | |--------|------------------| | Update SSH Keys | No | | Convert to Committed | No — billing only, nothing on the cluster changes | | Slurm Configuration | No — running jobs continue; queued jobs use the new settings | | Reconfigure Cluster | No | | Scale up | No | | Mount / unmount a volume | Yes — the nodes restart | | Update Image | Yes, gracefully — nodes are drained one at a time | | Restart Login Service | Only shells on the login node; Slurm jobs keep running | | Restart All Workers | **Yes — all running jobs** | | Restart Cluster | **Yes — all running jobs** | | Scale down | **Yes** — on removed nodes, and remaining workers restart too | | Terminate / Delete | **Yes — everything** | --- ## Permissions Actions are gated by your role on **Training Clusters** in the project. | You need | For | |----------|-----| | **Read** | Viewing clusters, all tabs, and **Connect** | | **Update** | Every action in the Actions menu except Clone and Delete; partitions; volumes; IPs; security groups; node reboot | | **Create** | **Clone Cluster** — it leads to a create form | | **Delete** | **Delete Training Cluster** | If the whole Actions menu is missing, or the details page is read-only, you have read access only. Ask a project administrator to review your role in the **IAM Panel**. --- ## Related Resources - [Cluster actions](/docs/tir/SlurmCluster/manage/actions) - [Manage a Slurm Cluster](/docs/tir/SlurmCluster/manage/) - [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) - [Troubleshooting](/docs/tir/SlurmCluster/troubleshoot/)