Skip to main content

Step by Step Guide to Pipeline

Introduction​

In the context of Artificial Intelligence (AI), a pipeline refers to a series of data processing steps or operations that are performed in sequence to achieve a specific AI task or goal. An AI pipeline typically involves several stages, each with a specific function, and it is designed to process and transform input data into meaningful output.

Each stage in the pipeline plays a crucial role in the overall AI process, and the effectiveness of the pipeline depends on the quality of data, the choice of algorithms, and the expertise in designing and optimizing each step. AI pipelines are commonly used in various applications, including machine learning, natural language processing, computer vision, and more.

What is a Pipeline?​

  • TIR Pipelines offer a way to write scalable, serverless, and asynchronous training jobs based on Docker containers. The supported formats include Argo and Kubeflow Pipelines Templates.
  • You no longer have to worry about the reliability of jobs, as TIR pipelines offer best-in-class retry functionality. This allows you to restart a job without losing completed work.
  • Additionally, TIR pipelines also support unlimited re-runs, stored results (in EOS buckets), and all resource plans (CPU and GPU).

How to Create a Pipeline​

To create the Pipeline, first, the user should navigate to the sidebar section and select Pipelines. Upon selecting Pipelines, a dropdown menu will appear, featuring an option labeled Pipeline.

Upon clicking the Pipeline option, the user will be directed to the Manage Pipelines page.

Manage Pipeline

Click on the "GET STARTED" button.

If you click on the "Get Started" button, the default pipeline will be created.

Click Get Started

Default Pipeline

Click on the "CREATE PIPELINE" button.

After being redirected to the "Manage Pipelines" page, users can locate and click on the CREATE PIPELINE button to create a pipeline.

Create Pipeline

After clicking the CREATE PIPELINE button, the Create Pipeline page will open. On this page, you can create a pipeline with two options: Create a new pipeline or Create a new pipeline version under an existing pipeline.

Create New Pipeline

Create a New Pipeline​

To create a new pipeline, you have to upload a .yaml file, provide a description, and click on the UPLOAD button.

Avoid introducing additional nodes or commands, as such additions could lead to conflicts between nodes within their YAML file.

New Pipeline Upload

After successfully creating the pipeline, you will see the screen below with pipeline versions and details. You can also create a Run and view existing Runs.

Created Pipeline

Create Run: For a selected pipeline, you can create a run by clicking the plus icon. You will be redirected to the Create Run page.

Click Create Run

View Runs: For a selected pipeline, you can view a run by clicking the arrow icon. This will redirect you to the Manage Run page where you can view created runs.

Click View Runs

Pipeline Versions​

You can see the pipeline versions by selecting a particular pipeline under the Pipeline Versions tab. You can create a version by clicking the CREATE VERSION button. After successfully creating pipeline versions, you can see the list.

View Pipeline Versions

Create Versions​

To create a new pipeline version for an existing pipeline, select the pipeline and then click Create Version under Pipeline Versions.

Create Version

After clicking Create Version, you will see the Create Pipeline screen. In that, you must upload a file of type .yaml and click on UPLOAD.

Pipeline Version Upload

You can view the created pipeline versions below.

View Pipeline Versions

Create Run​

You can create a run for a specific pipeline version. For that, you need to click on Create Run.

Create Run for Version

After clicking on Create Run, you will see the Create Run screen. In this, you can fill in the run details and click on Finish.

Run Details

View Run​

After successfully creating a run for a specific pipeline version, you can view that run by clicking View Runs.

View Run for Version

Pipeline Versions Actions​

You can delete a pipeline version by clicking the delete icon.

Delete Pipeline Version

Actions​

Create Version​

To create versions for a particular pipeline, select the pipeline, then click Create Version under Actions.

Actions Create Version

Delete​

To delete a pipeline, select the pipeline, then click Delete under Actions.

Actions Delete

After clicking the delete icon, you will see the Delete Pipeline popup. Then, click on the DELETE button.

Delete Pipeline Popup

Create a New Pipeline Version under an Existing Pipeline​

To create a new pipeline version under an existing pipeline, click on Create a new pipeline version under an existing pipeline. Then, select the pipeline in which you want to upload a new version, and upload the .yaml file. Click on the UPLOAD button.

Create New Version for Existing Pipeline

After successfully creating the version, you will see the pipeline versions under the Pipeline Versions tab.

View Versions