Storage and Volumes
The Volumes tab manages the filesystems attached to your cluster. Every volume you mount appears at the same path on the login node and every worker node, which is what makes distributed training possible.
The Storage Types
The tab has a sub-navigation on the left: Datasets, Shared File System, and Parallel File System — plus Weka in regions where it is offered.
| Type | Read/write | Required at create | Use it for |
|---|---|---|---|
| Parallel File System (PFS) | Read-write | Yes in most regions | High-throughput training data, checkpoints, container squash files |
| Shared File System (SFS) | Read-write | Optional | General shared storage: scripts, logs, home directories |
| Datasets | Read-only | Optional | Curated input data you never write to |
| Weka | Read-write | Yes where offered, in place of PFS | High-performance shared storage |
The console describes them as:
Datasets — Manage datasets attached to this training cluster. Datasets provide persistent storage for your training data.
Shared File System (SFS) — provides simple, scalable, and highly available file storage for cloud-based applications. It allows multiple nodes to access the same file system concurrently.
Parallel File System (PFS) — is optimized for high-performance computing environments, enabling simultaneous access to files by multiple processes across distributed nodes. It delivers rapid data throughput by distributing file data across multiple storage devices.
A dataset mount is read-only. A job that tries to write a checkpoint or a log there fails with a permission error. Checkpoints and logs need PFS, SFS or Weka.
The tables
Each sub-tab has Mounted and Unmounted views.
| Column | Shows |
|---|---|
| Name | The volume |
| Size | Its capacity |
| Storage Status | Whether the volume itself is ready (PFS and Weka) |
| Encryption / Encryption Type | Encryption state (Datasets) |
| Mount Status | Mounted or Unmounted on this cluster |
| Mount Path | The path when mounted; an input when not |
| Actions | Mount or Unmount |
Mount a Volume
-
Open the cluster → Volumes → the sub-tab for the type.
-
Switch to the Unmounted view.
-
Type a Mount Path in the row, e.g.
/pfsor/shared. -
Click Mount.
-
The dialog — e.g. Mounting SFS — warns:
Mounting SFS will require restarting the cluster nodes.
Do you wish to continue?
-
Click Continue.
This is the part people miss. The dialog says "restarting the cluster nodes" and it means every worker. All running jobs are lost.
Check the Jobs tab first and let jobs finish, or plan the mount for a quiet window. Better still, mount everything you expect to need at create time, when it costs nothing.
Mount path rules
| Rule | Detail |
|---|---|
Must start with / | pfs is invalid; /pfs is valid |
Cannot be / | The root itself is not allowed |
No // and no trailing / | /a//b and /pfs/ are invalid |
| No spaces | The space key is blocked in the field |
| Must be unique | Two volumes cannot share a path |
| Errors | Mount path is required · Invalid path format |
/pfs, /shared, /data beat /mnt/volume1. Keep them consistent across clusters so job scripts
port unchanged — and remember the path is baked into every script, #SBATCH --output line and
--container-mounts flag your team writes.
Unmount a Volume
- Volumes → the sub-tab → Mounted view.
- Click Unmount on the row.
- Confirm the dialog — e.g. Unmounting PFS — which carries the same restart warning.
The data is not deleted. Unmounting detaches the volume from this cluster; the volume and its contents remain and can be mounted elsewhere.
When unmount is blocked
| Situation | What you see |
|---|---|
| It is the last mounted volume | Cannot remove storage. At least one storage volume must remain mounted on the cluster. Mount a replacement first |
| It holds the persistent-login home directories | The dialog reads "This volume is in use by Persistent Login User Management. Disable it first." and offers only Cancel |
For the second case, disable Login User Management first — which preserves the home directories.
Storage Inside a Container
A container job starts from the image's own filesystem. Your mounts exist on the node, but you have to pass in what the job needs:
srun --container-image=/pfs/images/pytorch.sqsh \
--container-mounts=/pfs:/pfs,/shared:/shared \
--container-workdir=/pfs/project \
python train.py
You can remap paths — --container-mounts=/pfs/datasets/imagenet:/data makes the dataset appear at
/data, which is convenient when your code expects a fixed location.
--container-mounts is the most common container-job mistakeThe job starts, runs, and reports that your dataset does not exist. Nothing is wrong with the mount — it was simply never passed into the container.
What Goes Where
| Data | Put it on | Why |
|---|---|---|
| Training datasets | PFS or Weka | Throughput matters most here |
| Checkpoints | PFS or SFS | Must survive node restarts and be readable by every rank |
| Job scripts | SFS or PFS | So they survive a login-node restart |
Job logs (--output) | PFS or SFS | Otherwise lost with the pod |
| Container squash files | PFS or SFS | Every node reads the same file — see Cache images |
| Python environments | PFS or SFS, or a container image | Reinstalling after every restart is a waste of time |
| Per-job scratch | /tmp on the node | Fast, node-local, and cleaned up by a Worker Epilog |
| Very fast shared memory | /dev/shm | RAM-backed, per node, wiped with the pod |
/tmp, /dev/shm and anything in /root are private to one pod and erased when it is recreated —
which happens on every restart, scale, image update and node reboot. In a multi-node job, a
checkpoint written to /tmp on rank 0 is invisible to every other rank.
If it matters, it goes on PFS, SFS or Weka.
A layout that works
/pfs/
├── datasets/ # input data
├── images/ # container squash files
├── project/ # code, job scripts
├── checkpoints/ # per-run subdirectories
├── logs/ # sbatch --output / --error
└── envs/ # virtualenvs, if you are not using containers
Then #SBATCH --output=/pfs/logs/%x-%j.out and
--container-image=/pfs/images/pytorch-25.09.sqsh are the same in every script your team writes.
Capacity and Billing
Volumes are billed independently of the cluster. Terminating a cluster unmounts your volumes but does not delete them — the charges continue. Review PFS, SFS and datasets under Storage in the sidebar when you clean up. See Billing.
Keep an eye on usage from the login node:
df -h /pfs /shared
du -sh /pfs/*
du -sh /pfs/checkpoints/*
When PFS fills up, checkpoint writes fail mid-run, container imports fail, and job output truncates — often with errors that look nothing like "disk full". Checkpoints and squash files are the usual culprits. Prune old runs on a schedule, or with a Controller Epilog.