Live Replication Between S3-Bucket-Folder to EOS-Bucket-Folder

Overview

This guide covers setting up live replication between an AWS S3 bucket folder and an EOS (E2E Object Storage) bucket folder using AWS Lambda.

Step 1. IAM Role Creation with S3 Full Access

Create IAM Role

1. Navigate to AWS IAM Console.
2. Click on Roles → Create Role.
3. Choose AWS Service → Lambda.
4. Attach the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

4. Name the role LambdaS3FullAccess and create it.
5. Attach this role to your Lambda function.

Step2. Create an AWS Lambda Function

1. Open AWS Lambda Console → Create Function.
2. Choose Author from scratch.
3. Set runtime to Python 3.10.
4. Select the IAM role created earlier.
5. Click Create Function.

Step3. Setup a Test Event

1. Open AWS Lambda Console → Select your function.
2. Click Test → Create new test event.
3. Use the following JSON:

{
  "Records": [
    {
      "eventSource": "aws:s3",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "bucket": {
          "name": "S3_Bucket-name"
        },
        "object": {
          "key": "Folder-name/Object"
        }
      }
    }
  ]
}

4. Save the test event.

Step4. Lambda Deployment with Dependencies

Prepare Lambda Package

mkdir -p lambda_package

cd lambda_package

pip install minio -t lambda_package/

cd lambda_package/

Add Function Code

vim lambda_function.py

Paste the function code:

import os
import io
import json
import boto3
from minio import Minio
from minio.error import S3Error


# AWS S3 Client
s3_client = boto3.client('s3')


# MinIO Credentials
MINIO_ENDPOINT = os.getenv("MINIO_ENDPOINT", "objectstore.e2enetworks.net")
MINIO_ACCESS_KEY = os.getenv("MINIO_ACCESS_KEY")
MINIO_SECRET_KEY = os.getenv("MINIO_SECRET_KEY")
MINIO_BUCKET = os.getenv("MINIO_BUCKET")
S3_BUCKET = os.getenv("S3_BUCKET")
S3_FOLDER = os.getenv("S3_FOLDER")
MINIO_FOLDER = os.getenv("MINIO_FOLDER")


# MinIO Client
minio_client = Minio(
   MINIO_ENDPOINT,
   access_key=MINIO_ACCESS_KEY,
   secret_key=MINIO_SECRET_KEY,
   secure=True
)


def list_minio_objects():
   """Get a list of objects only within the MinIO sync folder."""
   objects = minio_client.list_objects(MINIO_BUCKET, prefix=f"{MINIO_FOLDER}/", recursive=True)
   return {obj.object_name for obj in objects}


def sync_folders():
   """Sync objects from S3 to MinIO and remove deleted objects only within the folder."""
  
   # Fetch objects in S3
   s3_objects = s3_client.list_objects_v2(Bucket=S3_BUCKET, Prefix=S3_FOLDER)
   s3_keys = set()


   if 'Contents' in s3_objects:
       for obj in s3_objects['Contents']:
           relative_path = obj['Key'][len(S3_FOLDER):].lstrip('/')
           s3_keys.add(relative_path)
           minio_object_path = f"{MINIO_FOLDER}/{relative_path}"


           # Check if the object exists in MinIO
           try:
               minio_client.stat_object(MINIO_BUCKET, minio_object_path)
               print(f"Skipping {minio_object_path} (Already exists)")
               continue
           except S3Error:
               pass  # Proceed with upload if file is missing


           # Stream S3 object into memory
           response = s3_client.get_object(Bucket=S3_BUCKET, Key=obj['Key'])
           data = io.BytesIO(response['Body'].read())


           # Upload directly to MinIO
           minio_client.put_object(MINIO_BUCKET, minio_object_path, data, length=len(data.getvalue()))
           print(f"Uploaded {obj['Key']} to MinIO as {minio_object_path}")


   # Find objects in MinIO **only within the sync folder** that do not exist in S3 and delete them
   minio_objects = list_minio_objects()
   s3_keys_with_prefix = {f"{MINIO_FOLDER}/{key}" for key in s3_keys}
   objects_to_delete = minio_objects - s3_keys_with_prefix


   for obj in objects_to_delete:
       minio_client.remove_object(MINIO_BUCKET, obj)
       print(f"Deleted {obj} from MinIO (No longer in S3)")


def lambda_handler(event, context):
   try:
       sync_folders()
   except Exception as e:
       print(f"Error: {e}")
       return {'statusCode': 500, 'body': json.dumps('Replication Failed')}
  
   return {'statusCode': 200, 'body': json.dumps('Replication Successful')}

zip -r ../lambda_function.zip

cd ..

ls

Step5. Upload to AWS Lambda

1. Open AWS Lambda Console.
2. Select your function → Click Upload from → .zip file.
3. Choose lambda_function.zip and click Deploy.

Step6. Set Environment Variables in AWS Lambda

1. Open AWS Lambda Console → Select your function.
2. Click Configuration → Environment Variables.
3. Click Edit and add:
   • MINIO_ACCESS_KEY = YOUR_ACCESS_KEY
   • MINIO_SECRET_KEY = YOUR_SECRET_KEY
   • MINIO_ENDPOINT = objectstore.e2enetworks.net
   • MINIO_BUCKET = E2E-BUCKET_NAME
   • S3_BUCKET = YOUR-S3-BUCKET-NAME
4. Click Save.

Step7. Configuring S3 Event Notification

1. Navigate to Amazon S3 Console.
2. Select the source bucket.
3. Click Properties → Event Notifications → Create Event Notification.
4. Configure:
   • Name: s3-e2e
   • Event types: Check all (Create, Remove, Restore).
   • Destination: Choose Lambda Function → Select your function.
5. Save changes.

Step8. Testing the Lambda Function

1. Click Test → Select the created test event.
2. Click Invoke.
3. The function should display Replication Successful if it works correctly.

Limitations & Workaround:

⚠ Existing Data Consideration – If you have any old data or manually uploaded objects in EOS, they will be deleted when the function runs.
⚠ However, this function only synchronizes objects inside the specified S3 folder and will not affect other bucket objects.
✅ Workaround: You can upload the EOS folder's old objects to the S3 folder—this will allow the function to track them as well.

Recommendation

We strongly advise testing the solution in a test environment first. Once validated, you can proceed with configuration for the production environment.