Copy files to and from an instance

You can copy files between your computer and an instance over SSH with scp, sftp, or rsync, or upload and download them in JupyterLab. Copy into the workspace at /home/jovyan: files there survive restarts and stops, and files elsewhere in the container are lost when it restarts. For data that is too large to copy, or that several instances need, mount it instead.

Before you begin

• An instance in the Running state. SSH and the app answer only while the instance is running.
• For scp, sftp, or rsync: SSH set up on the instance, which means your SSH key attached and a security group that allows port 22. See Connect to an instance over SSH.
• For uploads in the browser: an instance whose image has JupyterLab.

Find the address to copy to

SSH Command, under Connection Details on the instance’s Overview tab, connects you as root to the instance’s address. The example commands use 203.0.113.10 as that address; replace it with the address in your SSH Command. Every command logs in as root on port 22.

In the commands, <private-key-file> is the path on your computer to the private key that matches the SSH key attached to the instance. Leave out -i <private-key-file> if your SSH client already uses that key.

Copy files with scp

To copy a file from your computer to the instance’s workspace, run this on your computer:

scp -i <private-key-file> <local-file> root@203.0.113.10:/home/jovyan/

Replace <local-file> with the path of the file on your computer. To copy a whole folder, add -r:

scp -r -i <private-key-file> <local-folder> root@203.0.113.10:/home/jovyan/

Downloading works the same way, with the two places swapped. This copies a file from the instance to the current folder on your computer:

scp -i <private-key-file> root@203.0.113.10:/home/jovyan/<remote-file> .

Replace <remote-file> with the file’s path inside /home/jovyan.

Copy files with sftp

sftp opens a session in which you can move several files and look around the instance as you go:

sftp -i <private-key-file> root@203.0.113.10

At the sftp> prompt, go to the workspace, upload with put, download with get, and leave with exit:

cd /home/jovyan
put <local-file>
get <remote-file>
exit

Copy large or changing folders with rsync

rsync copies only what has changed since the last copy, so it suits large folders and resuming an interrupted transfer. You need it on both your computer and the instance. scp and sftp are on every TIR pre-built image. rsync comes with the TensorFlow and Transformers images. On the Jupyter and PyTorch images, install it yourself as root; on an Ubuntu-based image, run:

apt-get update && apt-get install -y rsync

Installed packages are lost when the instance restarts, so install rsync again after a restart, or from a start script. Then, on your computer, copy a folder into the workspace:

rsync -avz -e "ssh -i <private-key-file>" <local-folder>/ root@203.0.113.10:/home/jovyan/<folder-name>/

Replace <local-folder> with the folder on your computer, and <folder-name> with the name it gets in the workspace. To copy the other way, swap the two places. Run the same command again to send only what changed.

Upload and download in JupyterLab

On an instance with JupyterLab, open the app with the launch button on the instance’s page. To upload, drag files from your computer into JupyterLab’s file browser, or use its upload button. To download a file, right-click it in the file browser and choose Download. Save uploads under /home/jovyan, so they survive restarts.

For files stored in Google Drive, see Access Google Drive from an instance.

Share data between instances

Rather than copying the same data to each instance, mount it on all of them:

• Mount the same Shared File System on each instance to read and write the same files.
• Mount the same dataset in an EOS bucket on each instance to read the same data. It is mounted read-only.
• Move a disk dataset from one instance to another by unmounting it from the first and mounting it on the second. It can be mounted on one instance at a time.

See Mount datasets and file systems on an instance.

Check the copy

Connect over SSH and list the workspace:

ssh -i <private-key-file> root@203.0.113.10 ls -lh /home/jovyan

The listing shows your files with their sizes. Files in the workspace stay through restarts and stops until you delete the instance; Where your files live on an instance covers which other folders keep theirs.

If a copy into /home/jovyan stops with no space left on device, the workspace is full. Increase it on the instance’s Workspace Size tab while the instance runs, then copy again. See Increase the workspace size. For a refused connection or a rejected key, see Troubleshooting TIR Instances.