---
title: Cache and Manage Container Images
sidebar_label: Cache images
---

import { Download, Archive, RefreshCw, Lock, Trash2 } from 'react-feather';

# Cache and Manage Container Images

Pulling a multi-gigabyte image on every job wastes time and hits registry rate limits. This page
covers the two ways to avoid that — **squash files on shared storage** and **named containers** — plus
private registries and cleanup.

<QuickNav items={[
{ href: '#the-problem-worth-solving', label: 'The problem', icon: <RefreshCw size={13} /> },
{ href: '#import-once-to-a-squash-file', label: 'Squash files', icon: <Archive size={13} /> },
{ href: '#named-containers', label: 'Named containers', icon: <Download size={13} /> },
{ href: '#private-registries', label: 'Private registries', icon: <Lock size={13} /> },
{ href: '#housekeeping', label: 'Housekeeping', icon: <Trash2 size={13} /> },
]} />

---

## The Problem Worth Solving

When you pass a registry reference to `--container-image`, Enroot imports the image on **each node
that runs a task**, into that node's own Enroot cache. That means:

- A 4-node job may pull the same image four times.
- A 16-task job array may pull it 16 times.
- A node restart or a rolling image update clears what was cached there.
- Registries rate-limit, and a large sweep can trip that limit mid-run.

:::note Registry pulls go out through the cluster's security group
The pull happens **from the node running the job**, so it is outbound traffic subject to the
[security group attached to the cluster](/docs/tir/SlurmCluster/manage/network-security#security-groups).
The default SSH group allows it; a group that restricts egress blocks pulls, and the failure looks
like a plain timeout. Importing once to a squash file avoids the question entirely.
:::

:::warning There is no automatic shared image cache
TIR does not configure a cluster-wide squash cache for you. Sharing an image across nodes and jobs is
something you set up, once, by putting a squash file on shared storage — the pattern below. If you
have read elsewhere that images are automatically converted and shared, that is not how this platform
behaves.
:::

### Find out where Enroot caches on your cluster

Enroot's cache and data paths come from the cluster image, and they are usually under your `$HOME`.
Check them on your own cluster before assuming:

```bash
ssh root@<cluster-ip>
```

```bash
cat /etc/enroot/enroot.conf 2>/dev/null
enroot start --help | head -40
echo "HOME=$HOME"
```

This matters because **`$HOME` differs depending on how you log in**:

| How you log in | Where `$HOME` lives | Consequence |
|----------------|---------------------|-------------|
| `ssh root@<ip>` (default) | Inside the login pod's own filesystem | Not shared with compute nodes; lost when the login service restarts |
| A [persistent login user](/docs/tir/SlurmCluster/connect/login-user-management) | On the shared filesystem you nominated | Shared with every node and durable across restarts |

Either way, the recommendation is the same: **store the artefacts you care about at an explicit path
on PFS/SFS**, not wherever a default happens to point.

---

## Import Once to a Squash File

This is the pattern to standardise on.

### 1. Import the image on the login node

```bash
ssh root@<cluster-ip>
```

```bash
mkdir -p /pfs/images
cd /pfs/images
enroot import -o pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3
```

`enroot import` pulls the layers and writes a single squashfs file. Expect this to take a few minutes
and a few GB.

```bash
ls -lh /pfs/images/
```

### 2. Point jobs at the squash file

```bash
srun --container-image=/pfs/images/pytorch-25.09.sqsh \
     --container-mounts=/pfs:/pfs \
     python train.py
```

Every node reads the same file from shared storage. No registry traffic, no per-node duplication, and
start-up is consistent.

:::tip Why this works on TIR
PFS, SFS and Weka volumes are mounted from the same filesystem on the login node **and** on every
worker, at the path you chose. A squash file written on the login node is immediately readable by
every node in the allocation — which is exactly what a shared image cache needs.
:::

### 3. Keep a naming convention

Squash files are opaque once you have a few. Encode the source tag in the filename:

```bash
/pfs/images/pytorch-25.09-py3.sqsh
/pfs/images/nemo-25.07.sqsh
/pfs/images/vllm-0.11.0.sqsh
/pfs/images/internal-trainer-v42.sqsh
```

And keep a note of provenance next to them:

```bash
cat >> /pfs/images/README.md <<'EOF'
pytorch-25.09-py3.sqsh   docker://nvcr.io/nvidia/pytorch:25.09-py3      imported 2026-09-10
internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42  imported 2026-09-10
EOF
```

:::danger Never write a squash file to a dataset mount
Dataset volumes are mounted **read-only**. Use PFS, SFS or Weka. Do not use `/tmp`, `/dev/shm` or any
node-local path — those are per-node and defeat the entire purpose.
:::

### Refreshing an image

Squash files do not update themselves. When the upstream tag moves, re-import to a new filename and
switch your job scripts over — that way an in-flight run is never disturbed:

```bash
enroot import -o /pfs/images/pytorch-25.11-py3.sqsh docker://nvcr.io/nvidia/pytorch:25.11-py3
```

---

## Named Containers

`--container-name` gives an imported container a name on that node, so subsequent steps and jobs on
the same node start instantly.

```bash
srun --container-image=/pfs/images/pytorch-25.09.sqsh \
     --container-name=pytorch \
     --container-mounts=/pfs:/pfs \
     python step_one.py

# Same node, same allocation — no re-import
srun --container-name=pytorch --container-mounts=/pfs:/pfs python step_two.py
```

| Use it for | Not for |
|------------|---------|
| Several steps in one job script | Sharing a container **between nodes** — the name is node-local |
| Repeated jobs landing on the same node | Long-term storage of a modified environment; use a squash file |

:::note A name is per node
A container created as `pytorch` on `slinky-0` does not exist on `slinky-1`. For multi-node jobs,
combine `--container-name` with a **squash file** as the image source so each node has a cheap local
name backed by the same shared artefact.
:::

You can also manage containers directly with the Enroot CLI:

```bash
enroot list                 # containers on this node
enroot create --name pytorch /pfs/images/pytorch-25.09.sqsh
enroot remove pytorch
```

---

## Private Registries

To pull from a registry that needs credentials, Enroot reads a per-user credentials file. The exact
location follows Enroot's configuration on your cluster — check it first:

```bash
cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config
```

The credentials file uses one `machine` line per registry:

```bash
mkdir -p ~/.config/enroot
cat > ~/.config/enroot/.credentials <<'EOF'
machine nvcr.io login $oauthtoken password <NGC_API_KEY>
machine registry.example.com login <username> password <token>
EOF
chmod 600 ~/.config/enroot/.credentials
```

Then import as usual:

```bash
enroot import -o /pfs/images/internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42
```

:::warning Credentials and shared home directories
If your `$HOME` is on shared storage — which it is for
[persistent login users](/docs/tir/SlurmCluster/connect/login-user-management) — then a credentials
file there is readable by every node under your identity. Keep it at mode `600`, use a scoped
read-only registry token rather than a personal password, and rotate it on the schedule your
organisation requires. Never put registry credentials in a job script, a
[prolog script](/docs/tir/SlurmCluster/slurm-configuration/prolog-epilog), or a
[lifecycle script](/docs/tir/SlurmCluster/getting-started/create-cluster#lifecycle-script-optional) — those
are stored with the cluster configuration.
:::

The recommended pattern is to **import once on the login node** with your credentials and have jobs
reference the resulting squash file. Job scripts then need no credentials at all.

:::info Kubernetes image pull secrets are unrelated
The cluster's own components (controller, workers, login node) are Kubernetes pods and use their own
image pull configuration. That has nothing to do with the registry your **job** containers come from.
:::

---

## Housekeeping

Squash files are large and shared storage is billed. Audit them periodically.

```bash
du -sh /pfs/images/*
```

```bash
# Remove a squash file no job references any more
rm /pfs/images/pytorch-24.03-py3.sqsh
```

```bash
# Clear a node-local Enroot container
enroot list
enroot remove <container-name>
```

| Check | Why |
|-------|-----|
| Old squash files | Each is often 5–20 GB; a year of tags adds up fast |
| Duplicate imports of the same tag | Usually two people importing independently — agree one shared path |
| `/pfs/images` growth against your PFS quota | A full filesystem fails jobs in confusing ways, including checkpoint writes |

:::tip Watch the storage, not just the cluster
Storage is billed separately from the cluster and **keeps billing after the cluster is terminated**.
See [Billing](/docs/tir/SlurmCluster/billing#storage-is-billed-separately).
:::

---

## Related Resources

- [Containers overview](/docs/tir/SlurmCluster/containers/)
- [Run containers in jobs](/docs/tir/SlurmCluster/containers/run-containers)
- [Multi-node container training](/docs/tir/SlurmCluster/containers/multi-node-training)
- [Troubleshoot container jobs](/docs/tir/SlurmCluster/troubleshoot/containers)
- [Storage and volumes](/docs/tir/SlurmCluster/manage/storage)