---
title: Storage and Volumes
sidebar_label: Storage
---

import { HardDrive, Database, Layers, Plus, AlertTriangle, Box } from 'react-feather';

# Storage and Volumes

The **Volumes** tab manages the filesystems attached to your cluster. Every volume you mount appears
at the same path on the login node **and every worker node**, which is what makes distributed
training possible.

<QuickNav items={[
{ href: '#the-storage-types', label: 'Storage types', icon: <HardDrive size={13} /> },
{ href: '#mount-a-volume', label: 'Mount a volume', icon: <Plus size={13} /> },
{ href: '#unmount-a-volume', label: 'Unmount a volume', icon: <AlertTriangle size={13} /> },
{ href: '#storage-inside-a-container', label: 'Inside a container', icon: <Box size={13} /> },
{ href: '#what-goes-where', label: 'What goes where', icon: <Database size={13} /> },
]} />

---

## The Storage Types

The tab has a sub-navigation on the left: **Datasets**, **Shared File System**, and
**Parallel File System** — plus **Weka** in regions where it is offered.

| Type | Read/write | Required at create | Use it for |
|------|------------|--------------------|------------|
| **Parallel File System (PFS)** | Read-write | **Yes** in most regions | High-throughput training data, checkpoints, container squash files |
| **Shared File System (SFS)** | Read-write | Optional | General shared storage: scripts, logs, home directories |
| **Datasets** | **Read-only** | Optional | Curated input data you never write to |
| **Weka** | Read-write | **Yes** where offered, in place of PFS | High-performance shared storage |

The console describes them as:

> **Datasets** — Manage datasets attached to this training cluster. Datasets provide persistent
> storage for your training data.
>
> **Shared File System (SFS)** — provides simple, scalable, and highly available file storage for
> cloud-based applications. It allows multiple nodes to access the same file system concurrently.
>
> **Parallel File System (PFS)** — is optimized for high-performance computing environments, enabling
> simultaneous access to files by multiple processes across distributed nodes. It delivers rapid data
> throughput by distributing file data across multiple storage devices.

:::danger Datasets cannot be written to
A dataset mount is read-only. A job that tries to write a checkpoint or a log there fails with a
permission error. Checkpoints and logs need PFS, SFS or Weka.
:::

### The tables

Each sub-tab has **Mounted** and **Unmounted** views.

| Column | Shows |
|--------|-------|
| **Name** | The volume |
| **Size** | Its capacity |
| **Storage Status** | Whether the volume itself is ready (PFS and Weka) |
| **Encryption** / **Encryption Type** | Encryption state (Datasets) |
| **Mount Status** | **Mounted** or **Unmounted** on this cluster |
| **Mount Path** | The path when mounted; an input when not |
| **Actions** | **Mount** or **Unmount** |

---

## Mount a Volume

1. Open the cluster → **Volumes** → the sub-tab for the type.
2. Switch to the **Unmounted** view.
3. Type a **Mount Path** in the row, e.g. `/pfs` or `/shared`.
4. Click **Mount**.
5. The dialog — e.g. **Mounting SFS** — warns:

   > Mounting SFS will require **restarting** the cluster nodes.
   >
   > Do you wish to continue?

6. Click **Continue**.

:::danger Mounting or unmounting restarts the cluster nodes, ending running jobs
This is the part people miss. The dialog says "restarting the cluster nodes" and it means every
worker. **All running jobs are lost.**

Check the [Jobs tab](/docs/tir/SlurmCluster/manage/jobs) first and let jobs finish, or plan the
mount for a quiet window. Better still, mount everything you expect to need
[at create time](/docs/tir/SlurmCluster/getting-started/create-cluster), when it costs nothing.
:::

### Mount path rules

| Rule | Detail |
|------|--------|
| Must start with `/` | `pfs` is invalid; `/pfs` is valid |
| Cannot be `/` | The root itself is not allowed |
| No `//` and no trailing `/` | `/a//b` and `/pfs/` are invalid |
| No spaces | The space key is blocked in the field |
| Must be unique | Two volumes cannot share a path |
| Errors | *Mount path is required* · *Invalid path format* |

:::tip Pick paths you will still understand in six months
`/pfs`, `/shared`, `/data` beat `/mnt/volume1`. Keep them consistent across clusters so job scripts
port unchanged — and remember the path is baked into every script, `#SBATCH --output` line and
`--container-mounts` flag your team writes.
:::

---

## Unmount a Volume

1. **Volumes** → the sub-tab → **Mounted** view.
2. Click **Unmount** on the row.
3. Confirm the dialog — e.g. **Unmounting PFS** — which carries the same restart warning.

The data is **not deleted**. Unmounting detaches the volume from this cluster; the volume and its
contents remain and can be mounted elsewhere.

### When unmount is blocked

| Situation | What you see |
|-----------|--------------|
| It is the last mounted volume | *Cannot remove storage. At least one storage volume must remain mounted on the cluster.* Mount a replacement first |
| It holds the persistent-login home directories | The dialog reads *"This volume is in use by Persistent Login User Management. **Disable** it first."* and offers only **Cancel** |

For the second case, disable
[Login User Management](/docs/tir/SlurmCluster/connect/login-user-management#disable-it) first — which
preserves the home directories.

---

## Storage Inside a Container

A [container job](/docs/tir/SlurmCluster/containers/) starts from the image's own filesystem. Your
mounts exist on the node, but you have to pass in what the job needs:

```bash
srun --container-image=/pfs/images/pytorch.sqsh \
     --container-mounts=/pfs:/pfs,/shared:/shared \
     --container-workdir=/pfs/project \
     python train.py
```

You can remap paths — `--container-mounts=/pfs/datasets/imagenet:/data` makes the dataset appear at
`/data`, which is convenient when your code expects a fixed location.

:::tip Forgetting `--container-mounts` is the most common container-job mistake
The job starts, runs, and reports that your dataset does not exist. Nothing is wrong with the mount —
it was simply never passed into the container.
:::

---

## What Goes Where

| Data | Put it on | Why |
|------|-----------|-----|
| Training datasets | PFS or Weka | Throughput matters most here |
| Checkpoints | PFS or SFS | Must survive node restarts and be readable by every rank |
| Job scripts | SFS or PFS | So they survive a login-node restart |
| Job logs (`--output`) | PFS or SFS | Otherwise lost with the pod |
| Container squash files | PFS or SFS | Every node reads the same file — see [Cache images](/docs/tir/SlurmCluster/containers/image-cache) |
| Python environments | PFS or SFS, or a container image | Reinstalling after every restart is a waste of time |
| Per-job scratch | `/tmp` on the node | Fast, node-local, and cleaned up by a [Worker Epilog](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog) |
| Very fast shared memory | `/dev/shm` | RAM-backed, per node, wiped with the pod |

:::danger Node-local paths are not shared and do not survive
`/tmp`, `/dev/shm` and anything in `/root` are private to one pod and erased when it is recreated —
which happens on every restart, scale, image update and node reboot. In a multi-node job, a
checkpoint written to `/tmp` on rank 0 is invisible to every other rank.

**If it matters, it goes on PFS, SFS or Weka.**
:::

### A layout that works

```
/pfs/
├── datasets/          # input data
├── images/            # container squash files
├── project/           # code, job scripts
├── checkpoints/       # per-run subdirectories
├── logs/              # sbatch --output / --error
└── envs/              # virtualenvs, if you are not using containers
```

Then `#SBATCH --output=/pfs/logs/%x-%j.out` and
`--container-image=/pfs/images/pytorch-25.09.sqsh` are the same in every script your team writes.

---

## Capacity and Billing

:::warning Storage is billed separately, and keeps billing after the cluster is gone
Volumes are billed independently of the cluster. **Terminating a cluster unmounts your volumes but
does not delete them** — the charges continue. Review PFS, SFS and datasets under **Storage** in the
sidebar when you clean up. See [Billing](/docs/tir/SlurmCluster/billing#storage-is-billed-separately).
:::

Keep an eye on usage from the login node:

```bash
df -h /pfs /shared
du -sh /pfs/*
du -sh /pfs/checkpoints/*
```

:::tip A full filesystem fails in confusing ways
When PFS fills up, checkpoint writes fail mid-run, container imports fail, and job output truncates —
often with errors that look nothing like "disk full". Checkpoints and squash files are the usual
culprits. Prune old runs on a schedule, or with a
[Controller Epilog](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog#worked-examples).
:::

---

## Related Resources

- [Parallel File System](/docs/tir/pfs/) · [Shared File System](/docs/tir/sfs/) · [Datasets](/docs/tir/Datasets/)
- [Cache and manage container images](/docs/tir/SlurmCluster/containers/image-cache)
- [Login User Management](/docs/tir/SlurmCluster/connect/login-user-management)
- [Create a Slurm Cluster](/docs/tir/SlurmCluster/getting-started/create-cluster)
- [Billing](/docs/tir/SlurmCluster/billing)