Upload data to a bucket

Upload files to a bucket from the portal, or from your own machine with an S3 tool when the data is large. Either way, the files land in the bucket and show on its Data Objects tab.

Before you begin

• A bucket; see Create a bucket.
• Permission to update datasets in the project. A Member gets it through a policy that allows updating datasets. You need update permission to upload, and read permission to open folders.
• For a bucket with User Managed encryption, the key. Use the commands in the Encryption section of the Setup tab to upload. See Create an encrypted bucket.

Choose how to upload

The portal uploads up to five files at a time, which suits a few files. For files over 5 GB the portal advises the MinIO CLI or s3cmd, which also suit many small files. One compressed archive uploads faster than thousands of small files. A single object can be as small as 0 bytes and as large as 5 TiB.

Upload from the portal

1. In the list, select the row of the bucket, then open the Data Objects tab.
2. Select Upload Data.
3. In Destination Bucket / Folder Path, keep / for the top level of the bucket, or type a folder path.
4. Select files, or drag files in. Any file type works, up to five files at a time.
5. Select Upload, and follow each file’s progress until it succeeds or fails.
6. When every file has finished, select Ok.

Upload with the MinIO CLI or s3cmd

Set up the tool with the commands on the Setup tab, as Connect to a bucket with the MinIO CLI or s3cmd describes. Then copy a folder with the Copy Folder to Bucket command, or upload one file with mc cp or s3cmd put.

Check the files

Refresh the list on the Data Objects tab. The files are listed under the folder you chose. A file that has yet to appear shows up after a refresh, or when you list the bucket with an S3 tool. If it is still missing, see Troubleshooting TIR Object Storage.

Your data is in the bucket. Mount the bucket on an instance to read it there, or browse, download, and delete objects.