Step by Step Guide to Pipeline

Introduction

In the context of Artificial Intelligence (AI), a pipeline refers to a series of data processing steps or operations that are performed in sequence to achieve a specific AI task or goal. An AI pipeline typically involves several stages, each with a specific function, and it is designed to process and transform input data into meaningful output. Each stage in the pipeline plays a crucial role in the overall AI process, and the effectiveness of the pipeline depends on the quality of data, the choice of algorithms, and the expertise in designing and optimizing each step. AI pipelines are commonly used in various applications, including machine learning, natural language processing, computer vision, and more

What is Pipeline

  • TIR Pipelines offer a way to write scalable, serverless and asynchronous training jobs based on docker containers. The supported formats include Argo and Kubeflow Pipelines Templates.

  • You no longer have to worry about the reliability of training of jobs as TIR pipelines offer best-in class retry function. This allows you to restart a job without losing completed work.

  • Additionally, TIR pipelines also support unlimited re-runs, stored results (in EOS buckets) and all resource plans (CPU and GPU).

How to create a Pipeline

To create the Pipeline, first, the user should navigate to the sidebar section and select “Pipelines”. Upon selecting Pipelines, a dropdown menu will appear, featuring an option labeled “Pipeline”.

Upon clicking the “Pipeline” option, the user will be directed to the “Manage Pipelines” page.

../_images/manage_pipeline.png

Click on “GET STARTED” button.

If you click on “Get started” button, the default pipeline will be created.

../_images/click_get_started.png ../_images/default_pipeline.png

Click on “CREATE PIPELINE” button.

After redirect to the “Manage Pipelines”, On this page, users can locate and click on the “CREATE PIPELINE” button to create Pipeline.

../_images/create_pipelinee.png

After clicking the ‘CREATE PIPELINE’ button, the ‘Create Pipeline’ page will open. On this page, you can create Pipeline with two options ‘Create a new pipeline’ or ‘Create a new pipeline version under an existing pipeline’.

../_images/create_pipeline_1.png

Create a new pipeline

To create a new pipeline, you have to upload a .yaml file and give a description and click on “UPLOAD” button. Avoid to introduce additional nodes or commands, as such additions could lead to conflicts between nodes within their YAML file.

../_images/create_new_pipeline.png

After successfully created pipeline, you can see the below screen with pipeline versions and pipeline details. You can also create Run and view Run.

../_images/created_pipeline.png

Create Run For a selected pipeline, you can create run by clicking the plus icon and you will be redirected to “Create Run” page.

../_images/click_create_run.png

View Runs For a selected pipeline, you can view run by clicking the arrow icon and you will be redirected to “Manage Run” page. In that page you can view the created runs.

../_images/click_view_runs.png

Pipeline Versions

You can see the Pipeline Versions of selecting particular pipeline under “Pipeline Versions” tab.You can create a version by clicking “CREATE VERSION” button.After successully created pipeline versions, you can see the list.

../_images/view_pipeline_versions.png

CREATE VERSIONS

To create a new pipeline version for an existing pipeline, select pipeline then click “Create Version” under pipeline Versions.

../_images/pipeline_versions_create_versions.png

After clicking Create Version, you Create a new pipeline, you will see create pipeline screen.In that you have to upload a file type of .yaml click on upload.

../_images/pipeline_versions_create_versions1.png

You can see the pipeline versions below.

../_images/pipeline_versions_create_versions2.png

Create Run

You can create a run for a particular for a particular pipeline versions. For that you have to click on Create Run.

../_images/pipeline_versione_create_run.png

After clicking on “Create Run “, you will see the “Create Run” screen. In that you can fill the run details and click on finish.

../_images/pipeline_ver_create_run.png

View Run

After successfully created Run for a particular pipeline versions, you can view that run by clicking “View Runs”.

../_images/pipeline_version_view_run.png

Pipeline Versions Actions

You can delete pipeline version by clicking delete icon.

../_images/pipeline_version_delete.png

Actions

Create Version

To create versions for a particular pipeline, select pipeline then click “Create Version” under Actions.

../_images/actions_create_versions.png

Delete

To delete pipeline, select pipeline then click “Delete” under Actions.

../_images/actions_delete.png

After clicking delete icon, you can see the Delete Pipeline popup then click on “DELETE” button.

../_images/actions_delete_pipeline.png

Create a new pipeline version under an existing pipeline

To create a new pipeline version under an existing pipeline click on “Create a new pipeline version under an existing pipeline” then select the pipeline in which you want to upload new version and upload .yaml file and click on “UPLOAD” button.

../_images/create_a_new_pipelineversion_under_an_existing_pipeline.png

After successfully created version, you can see the pipeline versions under “Pipeline Versions” tab.

../_images/view_pipeline_versions.png

CREATE VERSIONS

After clicking Create Version, you Create a new pipeline, you will see create pipeline screen.In that you have to upload a file type of .yaml click on upload.

../_images/click_button_pipeline_versions_create_versions1.png ../_images/pipeline_versions_create_versions1.png

You can see the pipeline versions below.

../_images/pipeline_versions_create_versions2.png

Create Run

You can create a run for a particular for a particular pipeline versions. For that you have to click on Create Run.

../_images/pipeline_versione_create_run.png

After clicking on “Create Run “, you will see the “Create Run” screen. In that you can fill the run details and click on finish.

../_images/pipeline_ver_create_run.png

View Run

After successfully created Run for a particular pipeline versions, you can view that run by clicking “View Runs”.

../_images/pipeline_version_view_run.png

Pipeline Versions Actions

You can delete pipeline version by clicking delete icon.

../_images/pipeline_version_delete.png

Actions

Create Version

To create versions for a particular pipeline, select pipeline then click “Create Version” under Actions.

../_images/actions_create_versions.png

Delete

To delete pipeline, select pipeline then click “Delete” under Actions.

../_images/actions_delete.png

After clicking delete icon, you can see the Delete Pipeline popup then click on “DELETE” button.

../_images/actions_delete_pipeline.png