Pipelines

Pipelines runs a workflow you describe in a YAML file. You upload the file as a pipeline, start a run of it on a CPU or GPU plan, and watch each step’s status and logs. A run can start once, or repeat every hour or every day on a schedule. The file is written in Argo Workflows format or compiled from the Kubeflow Pipelines SDK. Each step runs a container, so any container image can be a step.

Where to find it

In TIR, open your project. In the side menu, under PRODUCTS, select Pipeline. The first time, you land on a page with Create Pipeline. After that, the Manage Pipelines screen lists your pipelines.

What you can do

Build and run

• Create a pipeline by uploading a YAML file.
• Create a pipeline version to change a pipeline and keep its earlier versions.
• Create a run with an experiment and a plan.
• Schedule a run to repeat every hour or every day.
• Run a Docker image as a pipeline step.
• Transfer data between EOS and PFS with ready-made pipeline files.

Watch and manage

• Watch a run and read its logs.
• Retry, terminate, or delete a run.
• Manage scheduled runs: turn them on and off, see the runs they started, or delete them.
• View a pipeline version: its step graph and its YAML.
• Delete a pipeline or a version.

Understand

• How pipelines, versions, runs, and experiments fit together.
• Billing and plans for pipeline runs.

Where to start

Start with Get started with Pipelines, which takes you from a YAML file to a run with logs. For help with any message, Troubleshooting Pipelines has the fixes, and the Pipelines FAQ has short answers.