TIR Slurm Cluster FAQ

For step-by-step fixes, see Troubleshooting TIR Slurm Cluster.

Billing and plans

Does a Slurm Cluster cost more when it is busy?

No. A Slurm Cluster is billed per worker node, per hour, at the plan’s rate, whether jobs are running or the nodes sit idle. Individual jobs, job arrays, and container jobs carry no charge of their own. See Billing and plans.

Can I pause a cluster to stop paying?

No. Terminating the cluster is the only way to stop compute billing, and an idle cluster is billed at the full rate. Terminate it between runs, and clone it later to get the same configuration back. See Terminate or delete a cluster and Clone a cluster.

Am I billed while my cluster is in Creating, or if it fails?

Once the create succeeds, billing is dated from the moment you submitted the form, so it includes any time spent in Creating. Compute is free for a cluster that ends up Failed. See Billing and plans.

Are storage and reserved IPs billed with the cluster?

No, they are billed separately. Terminating the cluster unmounts its volumes and keeps them, and they keep billing until you delete them under Storage. See Terminate or delete a cluster.

Is a cluster on a TIR Private Cluster billed by the hour?

No. A Slurm Cluster built on a TIR Private Cluster draws its nodes from that cluster’s reserved pool and is billed as part of your private cluster plan. See Billing and plans.

What does converting to a committed plan charge?

You pay for the whole term at once, for every node. The dialog quotes the per-node price, and you pay that price multiplied by the node count. The cluster keeps running the whole time. See Convert a cluster to committed.

Can I scale a committed cluster?

No. A committed cluster keeps its node count for the whole term, and Scale Cluster is hidden on it. Size the cluster on On-Demand before you commit. See Scale a cluster.

Can I terminate a committed cluster before the term ends?

Yes. Terminate Cluster and Delete Cluster are available on a committed cluster. The price you paid at the start covers the whole term, so nothing is refunded for the days left. See Billing and plans.

Regions, sizes, and limits

Which regions offer Slurm Cluster?

Delhi and Chennai. A Parallel File System is offered in Delhi; in Chennai, create the cluster with a Shared File System or a dataset. See Create a Slurm Cluster.

What is the smallest cluster I can create?

One node with 8 GPUs. Every GPU plan is a whole node with 8 GPUs, and Workers starts at 1. See Create a Slurm Cluster.

How many nodes can a cluster have?

As many as there are free nodes of its type. The Scale Cluster dialog accepts a total of up to 512 nodes, and a scale-up needs free nodes on the same rack as the cluster’s existing nodes. See Scale a cluster.

Why does a GPU plan show Inventory not available?

Every node of that type is taken at the moment, and the same shortage can make a large cluster fail to create. Retry later, ask for fewer nodes, or reserve capacity with a TIR Private Cluster. See Create a Slurm Cluster.

How many partitions can I add?

Up to 20 of your own, on top of the platform’s all and slinky partitions. See Manage partitions.

How large can my slurm.conf settings and scripts be?

The extra slurm.conf block must stay under 8 KB, and each prolog or epilog script under 16 KB. See Change the Slurm configuration.

Up to 51 active members. TIR assigns each member a UID from 1000 to 1050, and when the range is full, TIR refuses a new member with “No available UID in range 1000–1050 for this project”. Delete a member to free a slot; that also deletes their home directory. A login name can be up to 32 characters. See Manage login users.

Access and logins

Who can change a cluster?

Anyone with Update permission on Training Clusters in the project. Read permission covers viewing and Connect. Update covers every action except cloning and deleting, plus partitions, volumes, IPs, security groups, and node reboot. Clone Cluster needs Create, and Delete Cluster needs Delete. A project administrator can change your role. See Cluster statuses and actions.

Can I change the root SSH keys on a cluster?

Yes, with Update SSH Keys, which replaces the whole set. It is offered in every status except Creating, Initializing Worker Nodes, Health Check Running, and Failed. To give a failed cluster’s setup new keys, clone it and choose the keys on the new form. See Update a cluster.

How do I see whose job is whose?

Turn on Login User Management, so each person has their own identity. When everyone logs in as root, the User column on the Jobs tab shows root for everyone. See Manage login users.

Yes. It restarts the cluster’s login service and controller, which drops open SSH sessions and anything running in a foreground shell on the login node. See Manage login users.

No. A login name belongs to one member for the life of the project, even after that member is deleted. Agree on a naming convention before you add members. See Manage login users.

Yes. Turning it off, revoking a group’s access, deleting a group, and terminating or deleting the cluster all keep home directories. Only deleting a member removes theirs. See Manage login users.

Jobs and software

Do I need to recreate the cluster to use a different PyTorch or CUDA version?

No. Pass --container-image to srun or sbatch. The cluster’s own image is its Slurm software stack; your job’s environment comes from the container image you name. See Run containers in jobs.

Can two teams use different framework versions on one cluster?

Yes. Each job names its own container image, so two teams can run different versions at the same time. See Run containers in jobs.

Can I cancel a job from the console?

No. The Jobs tab is read-only. Run scancel <jobid> over SSH. See See jobs and logs.

What if Slurm rejects a request for one GPU?

Ask for whole nodes instead. If srun --gres=gpu:1 answers Invalid generic resource (gres) specification, use --nodes=1 --exclusive, which gives the job every GPU on the node. Raise a support request if you need per-GPU allocation. See Troubleshooting jobs on TIR Slurm Cluster.

Does a Slurm Cluster have InfiniBand or RDMA?

Yes. The worker nodes have an RDMA fabric between them. A container job reaches it once you mount /dev/infiniband into the container. See Run multi-node training.

Why do the Monitoring charts show a different time from my job’s log?

The chart times are in India Standard Time, whatever the time zone of your computer. Allow for it when you compare a chart with your own timestamps. See Monitor a cluster.

Keep your work safe

Where should I write checkpoints and logs?

On a Parallel File System or Shared File System mount, which every node sees at the same path and which is kept across restarts. The container’s own filesystem is discarded at the end of the step, and node-local paths are erased by a restart and stay on their own node. See Mount and unmount volumes.

What erases a node’s own disk?

These actions recreate nodes: restarting the workers, the login service, or the whole cluster, scaling down, updating the image, mounting or unmounting a volume, and Reconfigure Cluster, which recreates the login node. Recreating a node erases /root, /tmp, /dev/shm, and anything you installed by hand. See How a Slurm Cluster works.

Does a node failure end my job?

Yes. A node failure, a preemption, or a restart ends the jobs on that node. Checkpoint often to shared storage and make your script resume from the latest checkpoint. The cluster itself recovers, and its job queue and history survive a controller restart. See How a Slurm Cluster works.

No. The login service runs as one copy, so a login node restart drops every open session. Submit long work with sbatch, which keeps running when your connection drops. See Connect to a cluster.

What should I check before a restart, scale-down, or termination?

The Jobs tab, or squeue. These actions go ahead whatever is in the queue and end the jobs in flight; only Update Image drains nodes first. Let jobs finish, or drain the nodes with scontrol update nodename=<node> state=drain reason="<why>". See Cluster statuses and actions.

Should I cache container images?

Yes. Import each image once to a squash file on shared storage and point jobs at the file, so each node and array task reads the file instead of pulling the image again. See Cache container images.

What happens if a worker prolog fails?

Slurm drains the node, and it stops taking jobs. A buggy prolog can take the whole cluster out of service one node per job, so end every worker prolog with exit 0. See Add prolog and epilog scripts.

Should short jobs get their own partition?

Yes, on a shared cluster. A partition with a one-hour Max time keeps quick tests and interactive work from waiting behind a long training run. See Manage partitions.

When everyone uses root, scancel -u root cancels every job on the cluster, and every job shows root as its owner. See Manage login users.

Settings and partitions

Can I add a partition while the cluster is being created?

No. A partition names specific nodes, and those nodes exist only once the cluster is created. Add it from the Cluster Overview tab once the cluster is Running. See Manage partitions.

Can I edit slurm.conf over SSH?

You can, but the next change of any kind to the cluster overwrites a hand edit. Make the change in the Slurm Configuration dialog, where it stays. See Change the Slurm configuration.

Why was my slurm.conf line rejected?

The dialog rejects a key the platform manages, a duplicate key, a line without =, a value with a space, #, ;, or a backslash, and a block over 8 KB. See Troubleshooting TIR Slurm Cluster.

Why does my GresTypes setting not take effect?

On a GPU cluster, the platform sets its own values for GresTypes, SelectTypeParameters, and AccountingStorageTRES. The dialog accepts your values for these three keys, but the platform’s are the ones the scheduler uses. Run scontrol show config | grep -i <key> on the login node to see them. See Change the Slurm configuration.

Nodes, actions, and recovery

Does scaling up interrupt jobs?

No. New nodes join while the existing nodes and their jobs keep running. Add the new nodes to your own partitions afterwards. See Scale a cluster.

Does scaling down interrupt jobs?

Yes. Scaling down ends jobs on the removed nodes and restarts the remaining workers, which ends jobs there too. It goes ahead whatever is in the queue. See Scale a cluster.

Will an image update interrupt my jobs?

Yes, one node at a time. The update drains each node before it restarts it, and the dialog estimates 3 to 8 minutes of downtime per node, so a long-running job is still affected. Schedule it. See Update a cluster.

Should I reboot one node or restart all workers?

Reboot the one node. Restart All Workers ends every running job. Escalate from Restart Login Service to Restart All Workers to Restart Cluster only as needed. See Check cluster and node health.

What does a node stuck Unknown mean?

Slurm has no report for the node. It is normal for a few minutes after a restart, a scale-up, or an image update. If it stays, check that worker’s log, then reboot the node. See Troubleshooting TIR Slurm Cluster.

What if a node stays down after a reboot?

Contact support. After Reboot Node, the node shows as rebooting for 5 minutes and then its real state, and TIR tries the reboot only once. Include the node’s machine identifier, which starts with e2e_node_, and that worker’s log. See Troubleshooting TIR Slurm Cluster.

Can I repair a Failed cluster?

No. Clone it to start a fresh cluster with the same configuration, then delete the failed record. See Clone a cluster.

Can I get a deleted cluster back?

No. Deleting is permanent: the cluster leaves the list for good. A terminated cluster stays in the list and can still be cloned, so terminate rather than delete while you might want the setup again. See Terminate or delete a cluster.

Check the cluster’s status first, then its plan, then your permission. Most actions need Running, Scale Cluster is hidden on a committed plan, and a read-only page means your role needs Update permission. See Cluster statuses and actions.

Why will my cluster not terminate?

A VPC IP that is still attaching blocks both terminate and delete. Wait for the attach to finish on the Network & Security tab, then try again. See Terminate or delete a cluster.

Can I mount a dataset or volume on a running cluster?

Yes, from the Volumes tab. Mounting or unmounting restarts the cluster’s worker nodes and login node, and jobs running on them are lost. See Mount and unmount volumes.

Why can I not unmount a volume?

A cluster must keep at least one volume mounted, so mount a replacement first. A volume that holds the Login User Management home directories stays mounted until you turn that feature off. See Mount and unmount volumes.

How do I contact support?

Raise a ticket from Support in the TIR sidebar, or email cloud-platform@e2enetworks.com. Give the cluster name and the project, plus the detail for your problem. See Troubleshooting TIR Slurm Cluster.