Setting up s3cmd with E2E Object Storage

Introduction

s3cmd is a popular cross-platform command-line tool for managing S3 and S3-compatible object stores.In this article we will show you how to configure and use S3cmd to manage data with E2E Object Storage

Prerequestie

1.Bucket created in E2E Objectstorage with necessary permission and Its access key pairs.If you have not yet created you can refer this article to get started up with object storage.

2.Linux server with Sudo access to install and configure s3cmd cli

Step 1 : Installing s3cmd on Linux server

s3cmd is available in default rpm repositories for CentOS, RHEL and Ubuntu systems, You can install it using simply executing following commands on your system.

On CentOS/RHEL

yum install s3cmd

On Ubuntu/Debian

sudo apt-get install s3cmd

Step 2 : Configuring s3cmd

In order to configure s3cmd you would require following details to enter,Kindly make sure to enter S3 Endpoint and DNS option as shown below while configuring s3cmd

Access Key  : xxxxxxxxxxxxxxxxxxxx
Secret Key  : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
S3 Endpoint : objectstore.e2enetworks.net
DNS-style bucket+hostname:port template for accessing a bucket : %(bucket)s.objectstore.e2enetworks.net

Once you have got your access key and Secret key,Run the the below command to configure s3cmd

s3cmd --configure

You will get an output as below,Enter the following above details and leave the rest as default.

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: xxxxxxxxxxxxxxxxxxxxx
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: objectstore.e2enetworks.net

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: % (bucket)s.objectstore.e2enetworks.net

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: xxxxxxxxxxxxxxxxxxxxx
  Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  Default Region: US
  S3 Endpoint: objectstore.e2enetworks.net
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.objectstore.e2enetworks.net
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

You can also manually edit the file /root/.s3cfg and add below text to work s3cmd with EOS buckets,Kindly replace access_key and secret_key with your credentials.

# Setup endpoint
host_base = objectstore.e2enetworks.net
host_bucket = %(bucket)s.objectstore.e2enetworks.net


# Setup access keys
access_key = xxxxxxxxxxxxxxxxxxxx
secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Step 3 : Test access with supplied credentials

You can now Test access with supplied credentials,here we will list the bucket named “e2e-test” which is currently created.

s3cmd ls s3://e2e-test

To move a local test file to the bucket “e2e-test” use below command

touch testingfile
s3cmd sync testingfile s3://e2e-test/
upload: 'test' -> 's3://e2e-test/testingfile'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

s3cmd ls s3://e2e-test
2019-11-22 12:51         0   s3://e2e-test/testingfile

Conclusion

You have now successfully configured s3cmd on your server and will now be able to manage data with E2E ObjectStorage service. The complete user guide on usage of s3cmd is here