---
title: "Transfer data between EOS and PFS"
description: "Copy a folder between an object storage bucket (EOS) and a Parallel File System (PFS) with two ready-made TIR pipeline files, once or on a schedule."
content_type: "how-to"
product: "pipelines"
platform: "TIR"
url: "/tir/pipelines/how-to/transfer-data-between-eos-and-pfs/"
topic: ["data"]
relationships: {"requires":["/tir/pipelines/how-to/create-a-pipeline/"]}
---

# Transfer data between EOS and PFS

Two pipeline files copy data between an object storage bucket (EOS) and a [Parallel File
System](/tir/tir-pfs/) (PFS). One copies a folder from a PFS to a bucket, and the other copies a
folder from a bucket to a PFS. Create a pipeline from the file you need, then create a run, once or
on a schedule, to move the data.

## Before you begin

- A bucket in **Storage** > **Object Storage (EOS Dataset)**. Open it and, on its **Overview** tab
  under **Storage Details**, find the **Bucket Name**, **Access Key**, and **Secret Key**. Use the
  **Bucket Name**, not the alias. The keys go into `bucket_access_key` and `bucket_secret_key`.
- A PFS in your project with status **OK**. A PFS is offered in Delhi, so these files work in a
  Delhi project. See [Create a PFS](/tir/tir-pfs/how-to/create-a-pfs/).

## Copy from a PFS to a bucket

The file mirrors `source_path` on the PFS into `destination_path` in the bucket, overwriting files
with the same name and retrying on errors. Save it as `fs-to-eos.yaml`.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-sync-fs-to-eos-
spec:
  arguments:
    parameters:
      - name: source_fs_name
        value: ""
      - name: source_path
        value: ""
      - name: destination_bucket_name
        value: ""
      - name: destination_path
        value: ""
      - name: bucket_endpoint_url
        value: https://objectstore.e2enetworks.net
      - name: bucket_access_key
        value: ""
      - name: bucket_secret_key
        value: ""
  entrypoint: data-sync-job
  volumes:
    - name: source-fs
      persistentVolumeClaim:
        claimName: "{{workflow.parameters.source_fs_name}}"
  templates:
    - name: data-sync-job
      metadata: {}
      container:
        name: mc-mirror
        image: minio/mc:RELEASE.2025-08-13T08-35-41Z
        command: ["/bin/sh", "-c"]
        args:
          - |
            echo "===== STARTING DATA-SYNC JOB ====="

            echo ">>> adding destination bucket..."
            mc alias set destination-bkt $MINIO_ENDPOINT_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY

            DATA_SOURCE="/source-fs/$SOURCE_PATH"
            DATA_DESTINATION="destination-bkt/$TARGET_BUCKET_NAME/$DESTINATION_PATH"

            # collapse multiple slashes into a single slash (in case concatenation causes //)
            DATA_SOURCE=$(echo "$DATA_SOURCE" | tr -s '/')
            DATA_DESTINATION=$(echo "$DATA_DESTINATION" | tr -s '/')

            echo ">>> Details:"
            echo ">>> datasync source:  $DATA_SOURCE"
            echo ">>> datasync destination:  $DATA_DESTINATION"

            echo ">>> datasync started ..."
            mc mirror $DATA_SOURCE  $DATA_DESTINATION  --overwrite --preserve --retry --summary
            echo ">>> datasync completed."

            echo "===== DATA-SYNC JOB COMPLETE! ====="
        env:
          - name: MINIO_ENDPOINT_URL
            value: "{{workflow.parameters.bucket_endpoint_url}}"
          - name: TARGET_BUCKET_NAME
            value: "{{workflow.parameters.destination_bucket_name}}"
          - name: MINIO_ACCESS_KEY
            value: "{{workflow.parameters.bucket_access_key}}"
          - name: MINIO_SECRET_KEY
            value: "{{workflow.parameters.bucket_secret_key}}"
          - name: SOURCE_PATH
            value: "{{workflow.parameters.source_path}}"
          - name: DESTINATION_PATH
            value: "{{workflow.parameters.destination_path}}"
        volumeMounts:
          - name: source-fs
            mountPath: /source-fs
      inputs: {}
      outputs: {}
```

| Parameter                 | What to enter                              |
| ------------------------- | ------------------------------------------ |
| `source_fs_name`          | The name of the PFS.                       |
| `source_path`             | The folder on the PFS to upload.           |
| `destination_bucket_name` | The bucket name.                           |
| `destination_path`        | The folder in the bucket to copy into.     |
| `bucket_endpoint_url`     | The EOS endpoint. The file already has it. |
| `bucket_access_key`       | Your EOS access key.                       |
| `bucket_secret_key`       | Your EOS secret key.                       |

## Copy from a bucket to a PFS

The file mirrors `source_path` in the bucket into `destination_path` on the PFS, with the same
overwrite and retry behaviour. Save it as `eos-to-fs.yaml`.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-sync-eos-to-fs-
spec:
  arguments:
    parameters:
      - name: source_bucket_name
        value: ""
      - name: source_path
        value: ""
      - name: bucket_endpoint_url
        value: https://objectstore.e2enetworks.net
      - name: bucket_access_key
        value: ""
      - name: bucket_secret_key
        value: ""
      - name: destination_fs_name
        value: ""
      - name: destination_path
        value: ""
  entrypoint: data-sync-job
  volumes:
    - name: destination-fs
      persistentVolumeClaim:
        claimName: "{{workflow.parameters.destination_fs_name}}"
  templates:
    - name: data-sync-job
      metadata: {}
      container:
        name: mc-mirror
        image: minio/mc:RELEASE.2025-08-13T08-35-41Z
        command: ["/bin/sh", "-c"]
        args:
          - |
            echo "===== STARTING DATA-SYNC JOB ====="

            echo ">>> adding source bucket..."
            mc alias set source-bkt $MINIO_ENDPOINT_URL $MINIO_ACCESS_KEY $MINIO_SECRET_KEY

            DATA_SOURCE="source-bkt/$SOURCE_BUCKET_NAME/$SOURCE_PATH"
            DATA_DESTINATION="/destination-fs/$DESTINATION_PATH"

            # collapse multiple slashes into a single slash (in case concatenation causes //)
            DATA_SOURCE=$(echo "$DATA_SOURCE" | tr -s '/')
            DATA_DESTINATION=$(echo "$DATA_DESTINATION" | tr -s '/')

            echo ">>> Details:"
            echo ">>> datasync source:  $DATA_SOURCE"
            echo ">>> datasync destination:  $DATA_DESTINATION"

            echo ">>> datasync started ..."
            mc mirror $DATA_SOURCE  $DATA_DESTINATION  --overwrite --preserve --retry --summary
            echo ">>> datasync completed."

            echo "===== DATA-SYNC JOB COMPLETE! ====="
        env:
          - name: MINIO_ENDPOINT_URL
            value: "{{workflow.parameters.bucket_endpoint_url}}"
          - name: SOURCE_BUCKET_NAME
            value: "{{workflow.parameters.source_bucket_name}}"
          - name: MINIO_ACCESS_KEY
            value: "{{workflow.parameters.bucket_access_key}}"
          - name: MINIO_SECRET_KEY
            value: "{{workflow.parameters.bucket_secret_key}}"
          - name: SOURCE_PATH
            value: "{{workflow.parameters.source_path}}"
          - name: DESTINATION_PATH
            value: "{{workflow.parameters.destination_path}}"
        volumeMounts:
          - name: destination-fs
            mountPath: /destination-fs
      inputs: {}
      outputs: {}
```

| Parameter             | What to enter                               |
| --------------------- | ------------------------------------------- |
| `source_bucket_name`  | The bucket name.                            |
| `source_path`         | The folder or object prefix in the bucket.  |
| `bucket_endpoint_url` | The EOS endpoint. The file already has it.  |
| `bucket_access_key`   | Your EOS access key.                        |
| `bucket_secret_key`   | Your EOS secret key.                        |
| `destination_fs_name` | The name of the PFS.                        |
| `destination_path`    | The folder on the PFS to store the data in. |

## Create the pipeline and the run

1. [Create a pipeline](/tir/pipelines/how-to/create-a-pipeline/) from the file. Fill in the
   parameters in the file now, or leave them for the run.
2. [Create a run](/tir/pipelines/how-to/create-a-run/). Under **Run Parameters**, check every value.
   The run uses the values you type on the form. Choose a CPU plan; copying files runs well on a CPU.
3. When the run succeeds, open the run's name and the step's **Logs** to read the transfer summary.

To repeat the copy, turn on **Enable Scheduled Run**; see [Schedule a
run](/tir/pipelines/how-to/schedule-a-run/).