Cache container images

Save a container image once as a squash file on shared storage, so jobs start from that file instead of pulling a multi-gigabyte image again. This saves start-up time and avoids registry rate limits.

When you pass a registry reference to --container-image, Enroot imports the image on each node that runs a task, into that node’s own cache. A four-node job can pull the same image four times, and a 16-task job array can pull it 16 times. A node restart or a rolling image update clears what was cached there, and a large sweep can trip a registry’s rate limit mid-run.

To share one copy across every node and job, write an Enroot squash file to a Parallel File System or Shared File System mount once. You set up this cache yourself.

Before you begin

• A cluster that is Running and a shell on its login node. See Connect to a cluster.
• A Parallel File System or Shared File System volume mounted on the cluster. These commands use /shared as the mount path; use the one you set.

Find where Enroot caches on your cluster

Enroot’s cache and credentials locations come from /etc/enroot/enroot.conf on the cluster image, and they can differ between image versions. Check that file on your own cluster instead of assuming a path:

cat /etc/enroot/enroot.conf 2>/dev/null
enroot start --help | head -40
echo "HOME=$HOME"

$HOME depends on how you log in:

How you log in	Where $HOME is	What follows
ssh root@<cluster-ip>	Inside the login node’s own filesystem.	Only the login node sees it, and it is lost when the login service restarts.
A login user	On the shared volume chosen for Login User Management.	Every node sees it, and it is kept across restarts.

Either way, store the files you want to keep at a path you choose on shared storage.

Import the image once to a squash file

On the login node, import the image:

mkdir -p /shared/images
cd /shared/images
enroot import -o pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3

enroot import pulls the image’s layers and writes one squash file. It takes a few minutes and a few gigabytes. Check the result:

ls -lh /shared/images/

Point your jobs at the file:

srun --container-image=/shared/images/pytorch-25.09.sqsh \
     --container-mounts=/shared:/shared \
     python train.py

Every worker node in the allocation can read a file imported on the login node straight away, because the mount is the same filesystem at the same path everywhere on the cluster. Jobs start with no registry traffic and no copy on each node, and start-up takes the same time on every run.

Keep squash files off dataset mounts, /tmp, and /dev/shm. A dataset mount is read-only, and the other two belong to one node, which defeats a shared cache.

Name your squash files

Once you have a few squash files, a bare name stops telling you what is inside. Put the source tag in the file name:

/shared/images/pytorch-25.09-py3.sqsh
/shared/images/nemo-25.07.sqsh
/shared/images/vllm-0.11.0.sqsh
/shared/images/internal-trainer-v42.sqsh

Keep a short note about where each came from next to them:

cat >> /shared/images/README.md <<'EOF'
pytorch-25.09-py3.sqsh    docker://nvcr.io/nvidia/pytorch:25.09-py3        imported <date>
internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42    imported <date>
EOF

Replace <date> with the day you imported the image.

Refresh an image

A squash file keeps the image as it was imported. When the upstream tag moves, import the new image to a new file name, then switch your job scripts over, so a run in progress carries on with the old file:

enroot import -o /shared/images/pytorch-25.11-py3.sqsh docker://nvcr.io/nvidia/pytorch:25.11-py3

Keep a container between steps

--container-name gives an imported container a name on that node, so later steps and jobs on the same node start at once:

srun --container-image=/shared/images/pytorch-25.09.sqsh \
     --container-name=pytorch \
     --container-mounts=/shared:/shared \
     python step_one.py

# The same node and allocation: no import this time
srun --container-name=pytorch --container-mounts=/shared:/shared python step_two.py

Use it for several steps in one job script, or for repeated jobs that land on the same node. A name belongs to one node: a container named pytorch on slinky-0 lives on slinky-0 alone. For a multi-node job, use a squash file as the image source and add --container-name, so each node has a cheap local name backed by the same shared file. To keep a modified environment for the long term, save it as a squash file rather than a name.

You can also manage containers directly with the Enroot command line:

enroot list                 # containers on this node
enroot create --name pytorch /shared/images/pytorch-25.09.sqsh
enroot remove pytorch

Pull from a private registry

To pull from a registry that needs credentials, Enroot reads a per-user credentials file with one machine line for each registry. Its location follows your cluster’s Enroot configuration, so check it first:

cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config

A typical file, in Enroot’s usual location:

mkdir -p ~/.config/enroot
cat > ~/.config/enroot/.credentials <<'EOF'
machine nvcr.io login $oauthtoken password <ngc-api-key>
machine registry.example.com login <username> password <token>
EOF
chmod 600 ~/.config/enroot/.credentials

Replace <ngc-api-key>, <username>, and <token> with your own registry credentials. Then import as usual:

enroot import -o /shared/images/internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42

Keep registry credentials in this file at mode 600, and keep them out of job scripts, prologs, and lifecycle scripts: the cluster stores all three with its configuration and carries them into clones. If your $HOME is on shared storage, which it is for a login user, every node can read that file under your identity. Use a scoped, read-only registry token instead of a personal password, and rotate it on your organisation’s schedule.

The simplest pattern is to import once on the login node with your credentials and have jobs name the squash file, so job scripts need no credentials at all. The pull runs from the node doing the import, as outbound traffic that the cluster’s security group controls. A restrictive group can therefore break docker:// pulls with a plain network timeout. Importing once on the login node means jobs no longer depend on registry access.

Tidy up shared storage

Squash files are large, and shared storage is billed. Check them now and then:

du -sh /shared/images/*

# Remove a squash file that no job uses any more
rm /shared/images/pytorch-24.03-py3.sqsh

# Clear a container on this node
enroot list
enroot remove <container-name>

Replace <container-name> with a name from enroot list.

Look for old squash files, which are often 5 to 20 GB each, and for duplicate imports of the same tag, usually from two people importing the same image on their own. Agree on one shared path. Watch the size of the volume too: a full filesystem fails jobs in confusing ways, including checkpoint writes.

Storage is billed separately from the cluster and keeps billing after you terminate the cluster. See Billing and plans.

With your images on shared storage, every node and every job reads the same copy. For job scripts that use the squash files, see Run containers in jobs and Run multi-node training. For image import failures, see Troubleshooting containers on TIR Slurm Cluster.