How pipelines, versions, runs, and experiments fit together

Pipelines has five parts: a pipeline, its versions, runs, experiments, and scheduled runs.

Pipeline and version

A pipeline is a workflow you upload as a .yaml file. TIR accepts Argo Workflows format, which can lay steps out in sequence or as a graph, and files compiled from the Kubeflow Pipelines SDK. Each step runs a container. The pipeline list shows the type as Argo, Kubeflow, or None. A file that contains containerSet is refused.

A pipeline has versions. Each upload is a version, and the first one is the default version. Make a new version instead of replacing a pipeline: you keep the earlier ones and a run names the version it uses. See Create a pipeline version.

Run and experiment

A run is one execution of a pipeline version, on one CPU or GPU plan. Its status moves from Pending to Running to Succeeded, or to Failed, Error, or Terminated; see Watch a run and read its logs. You can retry a failed run and terminate a running one; see Retry, terminate, or delete a run.

An experiment is a name you attach each run to. It keeps a record of the pipelines and runs started under it, so you can group runs by project phase or model type. Every run needs one. You create experiments on the run form; the portal has no screen to list or delete them.

Scheduled run

A scheduled run starts a run again every hour or every day, from a start time to an end time you choose. Each run it starts is billed like any other. See Schedule a run.

Container images and data

A step can run any container image, public or private; see Run a Docker image. Two ready-made pipeline files move data between an object storage bucket and a Parallel File System; see Transfer data between EOS and PFS. When you create the first pipeline in a project, TIR sets up an object storage bucket and an access key for the project. Run artifacts are kept in that bucket.

Working well

• Keep YAML definitions simple, and avoid extra steps or commands that clash with steps already in the file.
• Use versions to track changes.
• Use experiments to group related runs.
• Use a CPU plan for steps that prepare or move data and a GPU plan for training. A run has one plan, so choose the plan the heaviest step needs; see Billing and plans for pipeline runs.
• Break a large pipeline into smaller steps and store intermediate results in an object storage bucket, so a failed step can be retried and reuse the finished results instead of computing everything again.
• Retry a failed run instead of starting over, and use a schedule for recurring batch work.
• Delete runs, versions, pipelines, and schedules once you are done with them.