Mount and unmount volumes

Add a storage volume to a running cluster, or detach one. A volume you mount appears at the same path on the login node and on every worker node, which is what makes distributed training work. Mounting or unmounting restarts the cluster’s nodes, so check what is running first.

Before you begin

• A cluster in the Running state. The Volumes tab opens only then.
• A volume to mount, in your project and in the cluster’s location: a Parallel File System, a Shared File System, or a dataset, which is an object storage bucket. Parallel File System is available only to accounts with that access, which for customers means Delhi.
• Running jobs finished, or safe to lose. Mounting or unmounting restarts the worker nodes and the login node whatever is in the Slurm queue, so jobs running on those nodes are lost. Check the Jobs tab, let jobs finish, or plan the change for a quiet period. Anything on a node’s own disk is erased as well.

Better still, mount everything you expect to need when you create the cluster, and you skip the restart later. See Create a Slurm Cluster.

Choose the type of volume

On the Volumes tab, the sub-navigation offers Datasets, Shared File System, and, for an account with Parallel File System access, Parallel File System.

Type	Writable	Use it for
Parallel File System	Yes, read-write.	Training data and checkpoints that many workers read and write at once.
Shared File System	Yes, read-write.	General shared storage: scripts, logs, checkpoints, and home directories.
Datasets	Read-only.	Curated input data that jobs only read, kept in object storage.

A job that tries to write a checkpoint or a log to a dataset mount fails with a permission error. Use a Parallel File System or Shared File System for anything the job writes. For how a Parallel File System and a Shared File System compare, see the TIR Parallel File System FAQ.

Each sub-tab has Mounted and Unmounted views. Their tables show Name, Size, the volume’s status, Mount Status, Mount Path, and a Mount or Unmount action.

Mount a volume

1. Open the cluster, select Volumes, then select the sub-tab for the type.
2. Switch to the Unmounted view.
3. In the volume’s row, type a Mount Path, for example /shared.
4. Select Mount. The dialog warns that mounting requires restarting the cluster nodes.
5. Select Continue.

A mount path must start with / and name a folder after it, use single slashes between parts, end with a name rather than /, and be free of spaces. It must also be unique on the cluster. TIR refuses a duplicate or conflicting path and names the path in the message. Choose paths you will still understand in six months: /shared and /data beat /mnt/volume1. The path ends up in every script, --output line, and --container-mounts flag your team writes, so keep it the same across clusters.

Once the nodes restart, the volume shows Mounted with its mount path.

Unmount a volume

1. Open the cluster, select Volumes, then select the sub-tab for the type.
2. Switch to the Mounted view.
3. Select Unmount on the volume’s row.
4. Confirm the dialog, which carries the same restart warning.

Unmounting detaches the volume from this cluster and keeps it. The volume and its contents remain, and you can mount it again here or on another cluster.

Two cases block an unmount:

• It is the last volume mounted on the cluster. A cluster must keep at least one storage volume mounted, so mount a replacement first.
• It holds the home directories for Login User Management. Turn that feature off first, which keeps the home directories. See Manage login users.

Use a mounted volume inside a container

A container job starts from the image’s own filesystem. Your mounts are on the node, and you pass in the ones the job needs:

srun --container-image=/shared/images/pytorch.sqsh \
     --container-mounts=/shared:/shared \
     --container-workdir=/shared/project \
     python train.py

You can remap a path: --container-mounts=/shared/datasets/imagenet:/data makes the dataset appear at /data. See Run containers in jobs.

Decide what goes where

Data	Put it on	Why
Training datasets	A Parallel File System or Shared File System	Many workers read them at once.
Checkpoints	A Parallel File System or Shared File System	They must survive node restarts and be readable by every rank.
Job scripts	A Shared File System or Parallel File System	They survive a login node restart.
Job logs (--output)	A Parallel File System or Shared File System	Logs on a node’s own disk are lost when the node is recreated.
Container squash files	A Parallel File System or Shared File System	Every node reads the same file. See Cache container images.
Python environments	A mounted volume, or a container image	You install once, and restarts leave them in place.
Per-job scratch	/tmp on the node	Fast and local to the node. A worker epilog can clean it up.
Fast memory-backed space	/dev/shm	It is RAM-backed, local to one node, and lost on restart.

/tmp, /dev/shm, and anything in /root belong to one node and are erased when it is recreated. That happens on every worker restart, login service restart, scale, image update, and volume mount or unmount. In a multi-node job, a checkpoint written to /tmp on rank 0 is visible on rank 0’s node only. If it matters, put it on a mounted volume.

A layout that works:

/shared/
├── datasets/          # input data
├── images/            # container squash files
├── project/           # code and job scripts
├── checkpoints/       # one subdirectory per run
├── logs/              # sbatch --output and --error
└── envs/              # virtual environments, if you are not using containers

With this layout, #SBATCH --output=/shared/logs/%x-%j.out and --container-image=/shared/images/pytorch-25.09.sqsh are the same in every script your team writes.

Watch capacity and cost

Volumes are billed independently of the cluster. Terminating a cluster unmounts its volumes and keeps them, so the charges continue until you delete the volumes yourself under Storage in the TIR sidebar. See Billing and plans.

Keep an eye on usage from the login node:

df -h /shared
du -sh /shared/*
du -sh /shared/checkpoints/*

A full filesystem fails in confusing ways. Checkpoint writes fail mid-run, container imports fail, and job output is cut short, often with errors that hide the real cause, a full disk. Old checkpoints and squash files are the usual cause, so remove old runs on a schedule, or with a controller epilog. See Add prolog and epilog scripts.

To read from your volumes in a job, see Get started with TIR Slurm Cluster. For a volume that fails to mount or unmount, see Troubleshooting TIR Slurm Cluster.