--- title: "TIR Slurm Cluster FAQ" description: "Short answers about TIR Slurm Cluster billing, regions, limits, logins, jobs, keeping work safe, settings, and recovery, each linking to the full page." content_type: "faq" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/faq/" --- # TIR Slurm Cluster FAQ For step-by-step fixes, see [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/). ## Billing and plans ### Does a Slurm Cluster cost more when it is busy? No. A Slurm Cluster is billed per worker node, per hour, at the plan's rate, whether jobs are running or the nodes sit idle. Individual jobs, job arrays, and container jobs carry no charge of their own. See [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/). ### Can I pause a cluster to stop paying? No. Terminating the cluster is the only way to stop compute billing, and an idle cluster is billed at the full rate. Terminate it between runs, and clone it later to get the same configuration back. See [Terminate or delete a cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/) and [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/). ### Am I billed while my cluster is in Creating, or if it fails? Once the create succeeds, billing is dated from the moment you submitted the form, so it includes any time spent in **Creating**. Compute is free for a cluster that ends up **Failed**. See [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/). ### Are storage and reserved IPs billed with the cluster? No, they are billed separately. Terminating the cluster unmounts its volumes and keeps them, and they keep billing until you delete them under **Storage**. See [Terminate or delete a cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/#delete-the-storage-you-no-longer-need). ### Is a cluster on a TIR Private Cluster billed by the hour? No. A Slurm Cluster built on a TIR Private Cluster draws its nodes from that cluster's reserved pool and is billed as part of your private cluster plan. See [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/#build-on-a-private-cluster). ### What does converting to a committed plan charge? You pay for the whole term at once, for every node. The dialog quotes the per-node price, and you pay that price multiplied by the node count. The cluster keeps running the whole time. See [Convert a cluster to committed](/tir/tir-slurm-cluster/how-to/convert-a-cluster-to-committed/). ### Can I scale a committed cluster? No. A committed cluster keeps its node count for the whole term, and **Scale Cluster** is hidden on it. Size the cluster on **On-Demand** before you commit. See [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/). ### Can I terminate a committed cluster before the term ends? Yes. **Terminate Cluster** and **Delete Cluster** are available on a committed cluster. The price you paid at the start covers the whole term, so nothing is refunded for the days left. See [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/#convert-to-a-committed-plan). ## Regions, sizes, and limits ### Which regions offer Slurm Cluster? Delhi and Chennai. A Parallel File System is offered in Delhi; in Chennai, create the cluster with a Shared File System or a dataset. See [Create a Slurm Cluster](/tir/tir-slurm-cluster/how-to/create-a-slurm-cluster/#add-storage). ### What is the smallest cluster I can create? One node with 8 GPUs. Every GPU plan is a whole node with 8 GPUs, and **Workers** starts at 1. See [Create a Slurm Cluster](/tir/tir-slurm-cluster/how-to/create-a-slurm-cluster/#choose-a-plan). ### How many nodes can a cluster have? As many as there are free nodes of its type. The **Scale Cluster** dialog accepts a total of up to 512 nodes, and a scale-up needs free nodes on the same rack as the cluster's existing nodes. See [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/). ### Why does a GPU plan show Inventory not available? Every node of that type is taken at the moment, and the same shortage can make a large cluster fail to create. Retry later, ask for fewer nodes, or reserve capacity with a [TIR Private Cluster](/tir/tir-private-cluster/). See [Create a Slurm Cluster](/tir/tir-slurm-cluster/how-to/create-a-slurm-cluster/). ### How many partitions can I add? Up to 20 of your own, on top of the platform's `all` and `slinky` partitions. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). ### How large can my slurm.conf settings and scripts be? The extra `slurm.conf` block must stay under 8 KB, and each prolog or epilog script under 16 KB. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). ### How many login users can a project have? Up to 51 active members. TIR assigns each member a UID from 1000 to 1050, and when the range is full, TIR refuses a new member with "No available UID in range 1000–1050 for this project". Delete a member to free a slot; that also deletes their home directory. A login name can be up to 32 characters. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). ## Access and logins ### Who can change a cluster? Anyone with Update permission on Training Clusters in the project. Read permission covers viewing and **Connect**. Update covers every action except cloning and deleting, plus partitions, volumes, IPs, security groups, and node reboot. **Clone Cluster** needs Create, and **Delete Cluster** needs Delete. A project administrator can change your role. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/#find-out-why-an-action-is-missing). ### Can I change the root SSH keys on a cluster? Yes, with **Update SSH Keys**, which replaces the whole set. It is offered in every status except **Creating**, **Initializing Worker Nodes**, **Health Check Running**, and **Failed**. To give a failed cluster's setup new keys, clone it and choose the keys on the new form. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/#update-ssh-keys). ### How do I see whose job is whose? Turn on Login User Management, so each person has their own identity. When everyone logs in as `root`, the **User** column on the **Jobs** tab shows `root` for everyone. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). ### Does turning on Login User Management interrupt anything? Yes. It restarts the cluster's login service and controller, which drops open SSH sessions and anything running in a foreground shell on the login node. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/#turn-on-login-user-management). ### Can I reuse a deleted member's login name? No. A login name belongs to one member for the life of the project, even after that member is deleted. Agree on a naming convention before you add members. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/#add-a-member). ### Are home directories kept when I turn Login User Management off? Yes. Turning it off, revoking a group's access, deleting a group, and terminating or deleting the cluster all keep home directories. Only deleting a member removes theirs. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/#what-a-members-home-directory-holds). ## Jobs and software ### Do I need to recreate the cluster to use a different PyTorch or CUDA version? No. Pass `--container-image` to `srun` or `sbatch`. The cluster's own image is its Slurm software stack; your job's environment comes from the container image you name. See [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/). ### Can two teams use different framework versions on one cluster? Yes. Each job names its own container image, so two teams can run different versions at the same time. See [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/). ### Can I cancel a job from the console? No. The **Jobs** tab is read-only. Run `scancel ` over SSH. See [See jobs and logs](/tir/tir-slurm-cluster/how-to/see-jobs-and-logs/). ### What if Slurm rejects a request for one GPU? Ask for whole nodes instead. If `srun --gres=gpu:1` answers `Invalid generic resource (gres) specification`, use `--nodes=1 --exclusive`, which gives the job every GPU on the node. Raise a support request if you need per-GPU allocation. See [Troubleshooting jobs on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/jobs/#invalid-generic-resource-gres-specification). ### Does a Slurm Cluster have InfiniBand or RDMA? Yes. The worker nodes have an RDMA fabric between them. A container job reaches it once you mount `/dev/infiniband` into the container. See [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/). ### Why do the Monitoring charts show a different time from my job's log? The chart times are in India Standard Time, whatever the time zone of your computer. Allow for it when you compare a chart with your own timestamps. See [Monitor a cluster](/tir/tir-slurm-cluster/how-to/monitor-a-cluster/). ## Keep your work safe ### Where should I write checkpoints and logs? On a Parallel File System or Shared File System mount, which every node sees at the same path and which is kept across restarts. The container's own filesystem is discarded at the end of the step, and node-local paths are erased by a restart and stay on their own node. See [Mount and unmount volumes](/tir/tir-slurm-cluster/how-to/mount-and-unmount-volumes/#decide-what-goes-where). ### What erases a node's own disk? These actions recreate nodes: restarting the workers, the login service, or the whole cluster, scaling down, updating the image, mounting or unmounting a volume, and **Reconfigure Cluster**, which recreates the login node. Recreating a node erases `/root`, `/tmp`, `/dev/shm`, and anything you installed by hand. See [How a Slurm Cluster works](/tir/tir-slurm-cluster/concepts/how-a-slurm-cluster-works/#what-is-kept-after-a-restart). ### Does a node failure end my job? Yes. A node failure, a preemption, or a restart ends the jobs on that node. Checkpoint often to shared storage and make your script resume from the latest checkpoint. The cluster itself recovers, and its job queue and history survive a controller restart. See [How a Slurm Cluster works](/tir/tir-slurm-cluster/concepts/how-a-slurm-cluster-works/#what-high-availability-covers). ### Is the login node highly available? No. The login service runs as one copy, so a login node restart drops every open session. Submit long work with `sbatch`, which keeps running when your connection drops. See [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/#submit-long-work-with-sbatch). ### What should I check before a restart, scale-down, or termination? The **Jobs** tab, or `squeue`. These actions go ahead whatever is in the queue and end the jobs in flight; only **Update Image** drains nodes first. Let jobs finish, or drain the nodes with `scontrol update nodename= state=drain reason=""`. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/#what-each-action-does-to-your-jobs). ### Should I cache container images? Yes. Import each image once to a squash file on shared storage and point jobs at the file, so each node and array task reads the file instead of pulling the image again. See [Cache container images](/tir/tir-slurm-cluster/how-to/cache-container-images/). ### What happens if a worker prolog fails? Slurm drains the node, and it stops taking jobs. A buggy prolog can take the whole cluster out of service one node per job, so end every worker prolog with `exit 0`. See [Add prolog and epilog scripts](/tir/tir-slurm-cluster/how-to/add-prolog-and-epilog-scripts/#write-a-safe-script). ### Should short jobs get their own partition? Yes, on a shared cluster. A partition with a one-hour **Max time** keeps quick tests and interactive work from waiting behind a long training run. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/#example-a-short-jobs-queue). ### Why should each person have their own login? When everyone uses `root`, `scancel -u root` cancels every job on the cluster, and every job shows `root` as its owner. See [Manage login users](/tir/tir-slurm-cluster/how-to/manage-login-users/). ## Settings and partitions ### Can I add a partition while the cluster is being created? No. A partition names specific nodes, and those nodes exist only once the cluster is created. Add it from the **Cluster Overview** tab once the cluster is **Running**. See [Manage partitions](/tir/tir-slurm-cluster/how-to/manage-partitions/). ### Can I edit slurm.conf over SSH? You can, but the next change of any kind to the cluster overwrites a hand edit. Make the change in the **Slurm Configuration** dialog, where it stays. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/). ### Why was my slurm.conf line rejected? The dialog rejects a key the platform manages, a duplicate key, a line without `=`, a value with a space, `#`, `;`, or a backslash, and a block over 8 KB. See [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#a-slurmconf-line-or-a-partition-is-rejected). ### Why does my GresTypes setting not take effect? On a GPU cluster, the platform sets its own values for GresTypes, SelectTypeParameters, and AccountingStorageTRES. The dialog accepts your values for these three keys, but the platform's are the ones the scheduler uses. Run `scontrol show config | grep -i ` on the login node to see them. See [Change the Slurm configuration](/tir/tir-slurm-cluster/how-to/change-the-slurm-configuration/#settings-the-platform-keeps). ## Nodes, actions, and recovery ### Does scaling up interrupt jobs? No. New nodes join while the existing nodes and their jobs keep running. Add the new nodes to your own partitions afterwards. See [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/). ### Does scaling down interrupt jobs? Yes. Scaling down ends jobs on the removed nodes and restarts the remaining workers, which ends jobs there too. It goes ahead whatever is in the queue. See [Scale a cluster](/tir/tir-slurm-cluster/how-to/scale-a-cluster/#scale-down). ### Will an image update interrupt my jobs? Yes, one node at a time. The update drains each node before it restarts it, and the dialog estimates 3 to 8 minutes of downtime per node, so a long-running job is still affected. Schedule it. See [Update a cluster](/tir/tir-slurm-cluster/how-to/update-a-cluster/#update-the-cluster-image). ### Should I reboot one node or restart all workers? Reboot the one node. **Restart All Workers** ends every running job. Escalate from **Restart Login Service** to **Restart All Workers** to **Restart Cluster** only as needed. See [Check cluster and node health](/tir/tir-slurm-cluster/how-to/check-cluster-and-node-health/#reboot-one-node). ### What does a node stuck Unknown mean? Slurm has no report for the node. It is normal for a few minutes after a restart, a scale-up, or an image update. If it stays, check that worker's log, then reboot the node. See [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#a-node-is-stuck-unknown). ### What if a node stays down after a reboot? Contact support. After **Reboot Node**, the node shows as rebooting for 5 minutes and then its real state, and TIR tries the reboot only once. Include the node's machine identifier, which starts with e2e_node_, and that worker's log. See [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#contact-support). ### Can I repair a Failed cluster? No. Clone it to start a fresh cluster with the same configuration, then delete the failed record. See [Clone a cluster](/tir/tir-slurm-cluster/how-to/clone-a-cluster/#replace-a-failed-cluster). ### Can I get a deleted cluster back? No. Deleting is permanent: the cluster leaves the list for good. A terminated cluster stays in the list and can still be cloned, so terminate rather than delete while you might want the setup again. See [Terminate or delete a cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/). ### Why is an action missing from the Actions menu? Check the cluster's status first, then its plan, then your permission. Most actions need **Running**, **Scale Cluster** is hidden on a committed plan, and a read-only page means your role needs Update permission. See [Cluster statuses and actions](/tir/tir-slurm-cluster/concepts/cluster-statuses-and-actions/#find-out-why-an-action-is-missing). ### Why will my cluster not terminate? A VPC IP that is still attaching blocks both terminate and delete. Wait for the attach to finish on the **Network & Security** tab, then try again. See [Terminate or delete a cluster](/tir/tir-slurm-cluster/how-to/terminate-or-delete-a-cluster/). ### Can I mount a dataset or volume on a running cluster? Yes, from the **Volumes** tab. Mounting or unmounting restarts the cluster's worker nodes and login node, and jobs running on them are lost. See [Mount and unmount volumes](/tir/tir-slurm-cluster/how-to/mount-and-unmount-volumes/). ### Why can I not unmount a volume? A cluster must keep at least one volume mounted, so mount a replacement first. A volume that holds the Login User Management home directories stays mounted until you turn that feature off. See [Mount and unmount volumes](/tir/tir-slurm-cluster/how-to/mount-and-unmount-volumes/#unmount-a-volume). ### How do I contact support? Raise a ticket from **Support** in the TIR sidebar, or email cloud-platform@e2enetworks.com. Give the cluster name and the project, plus the detail for your problem. See [Troubleshooting TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/tir-slurm-cluster/#contact-support).