Cache and Manage Container Images
Pulling a multi-gigabyte image on every job wastes time and hits registry rate limits. This page covers the two ways to avoid that — squash files on shared storage and named containers — plus private registries and cleanup.
The Problem Worth Solving
When you pass a registry reference to --container-image, Enroot imports the image on each node
that runs a task, into that node's own Enroot cache. That means:
- A 4-node job may pull the same image four times.
- A 16-task job array may pull it 16 times.
- A node restart or a rolling image update clears what was cached there.
- Registries rate-limit, and a large sweep can trip that limit mid-run.
The pull happens from the node running the job, so it is outbound traffic subject to the security group attached to the cluster. The default SSH group allows it; a group that restricts egress blocks pulls, and the failure looks like a plain timeout. Importing once to a squash file avoids the question entirely.
TIR does not configure a cluster-wide squash cache for you. Sharing an image across nodes and jobs is something you set up, once, by putting a squash file on shared storage — the pattern below. If you have read elsewhere that images are automatically converted and shared, that is not how this platform behaves.
Find out where Enroot caches on your cluster
Enroot's cache and data paths come from the cluster image, and they are usually under your $HOME.
Check them on your own cluster before assuming:
ssh root@<cluster-ip>
cat /etc/enroot/enroot.conf 2>/dev/null
enroot start --help | head -40
echo "HOME=$HOME"
This matters because $HOME differs depending on how you log in:
| How you log in | Where $HOME lives | Consequence |
|---|---|---|
ssh root@<ip> (default) | Inside the login pod's own filesystem | Not shared with compute nodes; lost when the login service restarts |
| A persistent login user | On the shared filesystem you nominated | Shared with every node and durable across restarts |
Either way, the recommendation is the same: store the artefacts you care about at an explicit path on PFS/SFS, not wherever a default happens to point.
Import Once to a Squash File
This is the pattern to standardise on.
1. Import the image on the login node
ssh root@<cluster-ip>
mkdir -p /pfs/images
cd /pfs/images
enroot import -o pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3
enroot import pulls the layers and writes a single squashfs file. Expect this to take a few minutes
and a few GB.
ls -lh /pfs/images/
2. Point jobs at the squash file
srun --container-image=/pfs/images/pytorch-25.09.sqsh \
--container-mounts=/pfs:/pfs \
python train.py
Every node reads the same file from shared storage. No registry traffic, no per-node duplication, and start-up is consistent.
PFS, SFS and Weka volumes are mounted from the same filesystem on the login node and on every worker, at the path you chose. A squash file written on the login node is immediately readable by every node in the allocation — which is exactly what a shared image cache needs.
3. Keep a naming convention
Squash files are opaque once you have a few. Encode the source tag in the filename:
/pfs/images/pytorch-25.09-py3.sqsh
/pfs/images/nemo-25.07.sqsh
/pfs/images/vllm-0.11.0.sqsh
/pfs/images/internal-trainer-v42.sqsh
And keep a note of provenance next to them:
cat >> /pfs/images/README.md <<'EOF'
pytorch-25.09-py3.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3 imported 2026-09-10
internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42 imported 2026-09-10
EOF
Dataset volumes are mounted read-only. Use PFS, SFS or Weka. Do not use /tmp, /dev/shm or any
node-local path — those are per-node and defeat the entire purpose.
Refreshing an image
Squash files do not update themselves. When the upstream tag moves, re-import to a new filename and switch your job scripts over — that way an in-flight run is never disturbed:
enroot import -o /pfs/images/pytorch-25.11-py3.sqsh docker://nvcr.io/nvidia/pytorch:25.11-py3
Named Containers
--container-name gives an imported container a name on that node, so subsequent steps and jobs on
the same node start instantly.
srun --container-image=/pfs/images/pytorch-25.09.sqsh \
--container-name=pytorch \
--container-mounts=/pfs:/pfs \
python step_one.py
# Same node, same allocation — no re-import
srun --container-name=pytorch --container-mounts=/pfs:/pfs python step_two.py
| Use it for | Not for |
|---|---|
| Several steps in one job script | Sharing a container between nodes — the name is node-local |
| Repeated jobs landing on the same node | Long-term storage of a modified environment; use a squash file |
A container created as pytorch on slinky-0 does not exist on slinky-1. For multi-node jobs,
combine --container-name with a squash file as the image source so each node has a cheap local
name backed by the same shared artefact.
You can also manage containers directly with the Enroot CLI:
enroot list # containers on this node
enroot create --name pytorch /pfs/images/pytorch-25.09.sqsh
enroot remove pytorch
Private Registries
To pull from a registry that needs credentials, Enroot reads a per-user credentials file. The exact location follows Enroot's configuration on your cluster — check it first:
cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config
The credentials file uses one machine line per registry:
mkdir -p ~/.config/enroot
cat > ~/.config/enroot/.credentials <<'EOF'
machine nvcr.io login $oauthtoken password <NGC_API_KEY>
machine registry.example.com login <username> password <token>
EOF
chmod 600 ~/.config/enroot/.credentials
Then import as usual:
enroot import -o /pfs/images/internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42
If your $HOME is on shared storage — which it is for
persistent login users — then a credentials
file there is readable by every node under your identity. Keep it at mode 600, use a scoped
read-only registry token rather than a personal password, and rotate it on the schedule your
organisation requires. Never put registry credentials in a job script, a
prolog script, or a
lifecycle script — those
are stored with the cluster configuration.
The recommended pattern is to import once on the login node with your credentials and have jobs reference the resulting squash file. Job scripts then need no credentials at all.
The cluster's own components (controller, workers, login node) are Kubernetes pods and use their own image pull configuration. That has nothing to do with the registry your job containers come from.
Housekeeping
Squash files are large and shared storage is billed. Audit them periodically.
du -sh /pfs/images/*
# Remove a squash file no job references any more
rm /pfs/images/pytorch-24.03-py3.sqsh
# Clear a node-local Enroot container
enroot list
enroot remove <container-name>
| Check | Why |
|---|---|
| Old squash files | Each is often 5–20 GB; a year of tags adds up fast |
| Duplicate imports of the same tag | Usually two people importing independently — agree one shared path |
/pfs/images growth against your PFS quota | A full filesystem fails jobs in confusing ways, including checkpoint writes |
Storage is billed separately from the cluster and keeps billing after the cluster is terminated. See Billing.