Skip to content

Transfer data between EOS and PFS

View as Markdown

Two pipeline files copy data between an object storage bucket (EOS) and a Parallel File System (PFS). One copies a folder from a PFS to a bucket, and the other copies a folder from a bucket to a PFS. Create a pipeline from the file you need, then create a run, once or on a schedule, to move the data.

Before you begin

Section titled “Before you begin”
  • A bucket in Storage > Object Storage (EOS Dataset). Open it and, on its Overview tab under Storage Details, find the Bucket Name, Access Key, and Secret Key. Use the Bucket Name, not the alias. The keys go into bucket_access_key and bucket_secret_key.
  • A PFS in your project with status OK. A PFS is offered in Delhi, so these files work in a Delhi project. See Create a PFS.

Copy from a PFS to a bucket

Section titled “Copy from a PFS to a bucket”

The file mirrors source_path on the PFS into destination_path in the bucket, overwriting files with the same name and retrying on errors. Save it as fs-to-eos.yaml.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-sync-fs-to-eos-
spec:
  arguments:
    parameters:
      - name: source_fs_name
        value: ""
      - name: source_path
        value: ""
      - name: destination_bucket_name
        value: ""
      - name: destination_path
        value: ""
      - name: bucket_endpoint_url
        value: https://objectstore.e2enetworks.net
      - name: bucket_access_key
        value: ""
      - name: bucket_secret_key
        value: ""
  entrypoint: data-sync-job
  volumes:
    - name: source-fs
      persistentVolumeClaim:
        claimName: "{{workflow.parameters.source_fs_name}}"
  templates:
    - name: data-sync-job
      metadata: {}
      container:
        name: mc-mirror
        image: minio/mc:RELEASE.2025-08-13T08-35-41Z
        command: ["/bin/sh", "-c"]
        args:
          - |
            echo "===== STARTING DATA-SYNC JOB ====="


            echo ">>> adding destination bucket..."
            mc alias set destination-bkt $MINIO_ENDPOINT_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY


            DATA_SOURCE="/source-fs/$SOURCE_PATH"
            DATA_DESTINATION="destination-bkt/$TARGET_BUCKET_NAME/$DESTINATION_PATH"


            # collapse multiple slashes into a single slash (in case concatenation causes //)
            DATA_SOURCE=$(echo "$DATA_SOURCE" | tr -s '/')
            DATA_DESTINATION=$(echo "$DATA_DESTINATION" | tr -s '/')


            echo ">>> Details:"
            echo ">>> datasync source:  $DATA_SOURCE"
            echo ">>> datasync destination:  $DATA_DESTINATION"


            echo ">>> datasync started ..."
            mc mirror $DATA_SOURCE  $DATA_DESTINATION  --overwrite --preserve --retry --summary
            echo ">>> datasync completed."


            echo "===== DATA-SYNC JOB COMPLETE! ====="
        env:
          - name: MINIO_ENDPOINT_URL
            value: "{{workflow.parameters.bucket_endpoint_url}}"
          - name: TARGET_BUCKET_NAME
            value: "{{workflow.parameters.destination_bucket_name}}"
          - name: MINIO_ACCESS_KEY
            value: "{{workflow.parameters.bucket_access_key}}"
          - name: MINIO_SECRET_KEY
            value: "{{workflow.parameters.bucket_secret_key}}"
          - name: SOURCE_PATH
            value: "{{workflow.parameters.source_path}}"
          - name: DESTINATION_PATH
            value: "{{workflow.parameters.destination_path}}"
        volumeMounts:
          - name: source-fs
            mountPath: /source-fs
      inputs: {}
      outputs: {}
Parameter What to enter
source_fs_name The name of the PFS.
source_path The folder on the PFS to upload.
destination_bucket_name The bucket name.
destination_path The folder in the bucket to copy into.
bucket_endpoint_url The EOS endpoint. The file already has it.
bucket_access_key Your EOS access key.
bucket_secret_key Your EOS secret key.

Copy from a bucket to a PFS

Section titled “Copy from a bucket to a PFS”

The file mirrors source_path in the bucket into destination_path on the PFS, with the same overwrite and retry behaviour. Save it as eos-to-fs.yaml.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-sync-eos-to-fs-
spec:
  arguments:
    parameters:
      - name: source_bucket_name
        value: ""
      - name: source_path
        value: ""
      - name: bucket_endpoint_url
        value: https://objectstore.e2enetworks.net
      - name: bucket_access_key
        value: ""
      - name: bucket_secret_key
        value: ""
      - name: destination_fs_name
        value: ""
      - name: destination_path
        value: ""
  entrypoint: data-sync-job
  volumes:
    - name: destination-fs
      persistentVolumeClaim:
        claimName: "{{workflow.parameters.destination_fs_name}}"
  templates:
    - name: data-sync-job
      metadata: {}
      container:
        name: mc-mirror
        image: minio/mc:RELEASE.2025-08-13T08-35-41Z
        command: ["/bin/sh", "-c"]
        args:
          - |
            echo "===== STARTING DATA-SYNC JOB ====="


            echo ">>> adding source bucket..."
            mc alias set source-bkt $MINIO_ENDPOINT_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY


            DATA_SOURCE="source-bkt/$SOURCE_BUCKET_NAME/$SOURCE_PATH"
            DATA_DESTINATION="/destination-fs/$DESTINATION_PATH"


            # collapse multiple slashes into a single slash (in case concatenation causes //)
            DATA_SOURCE=$(echo "$DATA_SOURCE" | tr -s '/')
            DATA_DESTINATION=$(echo "$DATA_DESTINATION" | tr -s '/')


            echo ">>> Details:"
            echo ">>> datasync source:  $DATA_SOURCE"
            echo ">>> datasync destination:  $DATA_DESTINATION"


            echo ">>> datasync started ..."
            mc mirror $DATA_SOURCE  $DATA_DESTINATION  --overwrite --preserve --retry --summary
            echo ">>> datasync completed."


            echo "===== DATA-SYNC JOB COMPLETE! ====="
        env:
          - name: MINIO_ENDPOINT_URL
            value: "{{workflow.parameters.bucket_endpoint_url}}"
          - name: SOURCE_BUCKET_NAME
            value: "{{workflow.parameters.source_bucket_name}}"
          - name: MINIO_ACCESS_KEY
            value: "{{workflow.parameters.bucket_access_key}}"
          - name: MINIO_SECRET_KEY
            value: "{{workflow.parameters.bucket_secret_key}}"
          - name: SOURCE_PATH
            value: "{{workflow.parameters.source_path}}"
          - name: DESTINATION_PATH
            value: "{{workflow.parameters.destination_path}}"
        volumeMounts:
          - name: destination-fs
            mountPath: /destination-fs
      inputs: {}
      outputs: {}
Parameter What to enter
source_bucket_name The bucket name.
source_path The folder or object prefix in the bucket.
bucket_endpoint_url The EOS endpoint. The file already has it.
bucket_access_key Your EOS access key.
bucket_secret_key Your EOS secret key.
destination_fs_name The name of the PFS.
destination_path The folder on the PFS to store the data in.

Create the pipeline and the run

Section titled “Create the pipeline and the run”
  1. Create a pipeline from the file. Fill in the parameters in the file now, or leave them for the run.
  2. Create a run. Under Run Parameters, check every value. The run uses the values you type on the form. Choose a CPU plan; copying files runs well on a CPU.
  3. When the run succeeds, open the run’s name and the step’s Logs to read the transfer summary.

To repeat the copy, turn on Enable Scheduled Run; see Schedule a run.

© E2E Networks Limited · CIN L72900DL2009PLC341980