Tutorial: Setting up s3cmd on Windows

s3cmd is a popular cross-platform command-line tool for managing S3-compatible object stores. In this tutorial, we will see how to configure and use S3cmd with E2E Object Storage on Windows server

Prerequisites

  1. Bucket in E2E Object Store. If you have not created a bucket yet, please refer here

  2. Access and Secret keys with permissions for the target bucket

  3. Windows system for installing s3cmd CLI

Download and install python 2.x

S3cmd requires python 2.7 or the higher version to run,You can install it here.Python 3 is currently not supported

After installation,Open the command terminal and add the python directory to your global PATH variable with below command.

setx PATH "%PATH%;C:\Python27\"

Note

Here python2.7 was installed,You need to change python version in above command as per your installation version.

Re-open the command terminal to utilize the new environment variable added above

Verify the installation with below command

C:\>python --version
Python 2.7.17

C:\>

Installing S3cmd

You can Download latest s3cmd source code from s3cmd official page and extract it to c:s3cmd location.

After extracting source code,navigate to the folder to which you downloaded s3cmd and run below command to setup

python setup.py install

Configure s3cmd

Please keep the below information handy as we will need it during the process.

Access Key  : xxxxxxxxxxxxxxxxxxxx
Secret Key  : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
S3 Endpoint : objectstore.e2enetworks.net
Bucket URL : %(bucket)s.objectstore.e2enetworks.net
Default Region: Leave blank (skip)

Go to the path where s3cmd was dowloaded

To start configuration in s3cmd, enter the following command:

python s3cmd --configure

Below is the snapshot of installation wizard. Follow the process in similar way to configure EOS with s3cmd CLI.

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: xxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: objectstore.e2enetworks.net

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: % (bucket)s.objectstore.e2enetworks.net

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Default Region: US
  S3 Endpoint: objectstore.e2enetworks.net
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.objectstore.e2enetworks.net
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n
Save settings? [y/N] y

Note

If you test the access with supplied credentials,you will get an error Test failed,Are you sure your keys have s3:ListallMybuckets permission You can ignore this error since you are using this with EOS bucket

Test access with configured credentials

List the contents of target bucket (eg. e2e-test) using a command like below. Please note the access and secret key that you had chosen during interactive/manual setup must have access to this bucket.Make sure you are on the path where s3cmd was downloaded.

python s3cmd ls s3://e2e-test

You may also test by moving local file to your target bucket (e.g. e2e-test) using below command

C:\s3cmd>python s3cmd sync "C:\e2ewindowstest.txt" s3://e2e-test/
upload: 'C:\e2ewindowstest.txt' -> 's3://e2e-test/e2ewindowstest.txt' (0 bytes in 0.0 seconds, -1.00 B/s) [1 of 1]

Conclusion

We have now successfully configured s3cmd to work with E2E Object Service. The complete user guide on usage of s3cmd is available here