--- title: "Cache container images" description: "Import a container image once to a squash file on shared storage so every node and every job on a TIR Slurm Cluster reads the same copy." content_type: "how-to" product: "tir-slurm-cluster" platform: "TIR" url: "/tir/tir-slurm-cluster/how-to/cache-container-images/" topic: ["run-jobs"] --- # Cache container images Save a container image once as a squash file on shared storage, so jobs start from that file instead of pulling a multi-gigabyte image again. This saves start-up time and avoids registry rate limits. When you pass a registry reference to `--container-image`, Enroot imports the image on each node that runs a task, into that node's own cache. A four-node job can pull the same image four times, and a 16-task job array can pull it 16 times. A node restart or a rolling image update clears what was cached there, and a large sweep can trip a registry's rate limit mid-run. To share one copy across every node and job, write an Enroot squash file to a Parallel File System or Shared File System mount once. You set up this cache yourself. ## Before you begin - A cluster that is **Running** and a shell on its login node. See [Connect to a cluster](/tir/tir-slurm-cluster/how-to/connect-to-a-cluster/). - A Parallel File System or Shared File System volume mounted on the cluster. These commands use `/shared` as the mount path; use the one you set. ## Find where Enroot caches on your cluster Enroot's cache and credentials locations come from `/etc/enroot/enroot.conf` on the cluster image, and they can differ between image versions. Check that file on your own cluster instead of assuming a path: ```bash cat /etc/enroot/enroot.conf 2>/dev/null enroot start --help | head -40 echo "HOME=$HOME" ``` `$HOME` depends on how you log in: | How you log in | Where `$HOME` is | What follows | | ----------------------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------- | | `ssh root@` | Inside the login node's own filesystem. | Only the login node sees it, and it is lost when the login service restarts. | | A [login user](/tir/tir-slurm-cluster/how-to/manage-login-users/) | On the shared volume chosen for Login User Management. | Every node sees it, and it is kept across restarts. | Either way, store the files you want to keep at a path you choose on shared storage. ## Import the image once to a squash file On the login node, import the image: ```bash mkdir -p /shared/images cd /shared/images enroot import -o pytorch-25.09.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3 ``` `enroot import` pulls the image's layers and writes one squash file. It takes a few minutes and a few gigabytes. Check the result: ```bash ls -lh /shared/images/ ``` Point your jobs at the file: ```bash srun --container-image=/shared/images/pytorch-25.09.sqsh \ --container-mounts=/shared:/shared \ python train.py ``` Every worker node in the allocation can read a file imported on the login node straight away, because the mount is the same filesystem at the same path everywhere on the cluster. Jobs start with no registry traffic and no copy on each node, and start-up takes the same time on every run. Keep squash files off dataset mounts, `/tmp`, and `/dev/shm`. A dataset mount is read-only, and the other two belong to one node, which defeats a shared cache. ## Name your squash files Once you have a few squash files, a bare name stops telling you what is inside. Put the source tag in the file name: ```bash /shared/images/pytorch-25.09-py3.sqsh /shared/images/nemo-25.07.sqsh /shared/images/vllm-0.11.0.sqsh /shared/images/internal-trainer-v42.sqsh ``` Keep a short note about where each came from next to them: ```bash cat >> /shared/images/README.md <<'EOF' pytorch-25.09-py3.sqsh docker://nvcr.io/nvidia/pytorch:25.09-py3 imported internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42 imported EOF ``` Replace `` with the day you imported the image. ## Refresh an image A squash file keeps the image as it was imported. When the upstream tag moves, import the new image to a new file name, then switch your job scripts over, so a run in progress carries on with the old file: ```bash enroot import -o /shared/images/pytorch-25.11-py3.sqsh docker://nvcr.io/nvidia/pytorch:25.11-py3 ``` ## Keep a container between steps `--container-name` gives an imported container a name on that node, so later steps and jobs on the same node start at once: ```bash srun --container-image=/shared/images/pytorch-25.09.sqsh \ --container-name=pytorch \ --container-mounts=/shared:/shared \ python step_one.py # The same node and allocation: no import this time srun --container-name=pytorch --container-mounts=/shared:/shared python step_two.py ``` Use it for several steps in one job script, or for repeated jobs that land on the same node. A name belongs to one node: a container named `pytorch` on `slinky-0` lives on `slinky-0` alone. For a multi-node job, use a squash file as the image source and add `--container-name`, so each node has a cheap local name backed by the same shared file. To keep a modified environment for the long term, save it as a squash file rather than a name. You can also manage containers directly with the Enroot command line: ```bash enroot list # containers on this node enroot create --name pytorch /shared/images/pytorch-25.09.sqsh enroot remove pytorch ``` ## Pull from a private registry To pull from a registry that needs credentials, Enroot reads a per-user credentials file with one `machine` line for each registry. Its location follows your cluster's Enroot configuration, so check it first: ```bash cat /etc/enroot/enroot.conf 2>/dev/null | grep -i config ``` A typical file, in Enroot's usual location: ```bash mkdir -p ~/.config/enroot cat > ~/.config/enroot/.credentials <<'EOF' machine nvcr.io login $oauthtoken password machine registry.example.com login password EOF chmod 600 ~/.config/enroot/.credentials ``` Replace ``, ``, and `` with your own registry credentials. Then import as usual: ```bash enroot import -o /shared/images/internal-trainer-v42.sqsh docker://registry.example.com/ml/trainer:v42 ``` Keep registry credentials in this file at mode 600, and keep them out of job scripts, prologs, and lifecycle scripts: the cluster stores all three with its configuration and carries them into clones. If your `$HOME` is on shared storage, which it is for a login user, every node can read that file under your identity. Use a scoped, read-only registry token instead of a personal password, and rotate it on your organisation's schedule. The simplest pattern is to import once on the login node with your credentials and have jobs name the squash file, so job scripts need no credentials at all. The pull runs from the node doing the import, as outbound traffic that the cluster's security group controls. A restrictive group can therefore break `docker://` pulls with a plain network timeout. Importing once on the login node means jobs no longer depend on registry access. ## Tidy up shared storage Squash files are large, and shared storage is billed. Check them now and then: ```bash du -sh /shared/images/* ``` ```bash # Remove a squash file that no job uses any more rm /shared/images/pytorch-24.03-py3.sqsh ``` ```bash # Clear a container on this node enroot list enroot remove ``` Replace `` with a name from `enroot list`. Look for old squash files, which are often 5 to 20 GB each, and for duplicate imports of the same tag, usually from two people importing the same image on their own. Agree on one shared path. Watch the size of the volume too: a full filesystem fails jobs in confusing ways, including checkpoint writes. Storage is billed separately from the cluster and keeps billing after you terminate the cluster. See [Billing and plans](/tir/tir-slurm-cluster/concepts/billing-and-plans/). With your images on shared storage, every node and every job reads the same copy. For job scripts that use the squash files, see [Run containers in jobs](/tir/tir-slurm-cluster/how-to/run-containers-in-jobs/) and [Run multi-node training](/tir/tir-slurm-cluster/how-to/run-multi-node-training/). For image import failures, see [Troubleshooting containers on TIR Slurm Cluster](/tir/tir-slurm-cluster/troubleshooting/containers/).