Kubeflow¶

Kubeflow is a free, open-source machine learning platform that allows machine learning pipelines to orchestrate complicated workflows running on Kubernetes.

For more detailed information regarding Kubeflow, please refer to the Kubeflow documentation.

Creating an Experiment¶

To create an experiment;

Open the DKubeX Terminal & clone the Kubeflow pipelines git repo using the command:
```
git clone https://github.com/kubeflow/pipelines.git
```
Click on the workspace’s terminal and create an experiment in the terminal using the command:
```
kfp experiment create <name>
```
Verify whether the experiment has been created using the following command:
```
kfp experiment list
```

Clone the Kubeflow pipelines git repo using the commands:

git clone https://github.com/kubeflow/pipelines.git

cd pipelines/samples/core/sequential/

Convert the .py file into a .yaml file to use the file in the pipeline.
```
kfp dsl compile --py <file_name>.py --output <file_name>.yaml
```

Create a pipeline using the above .yaml file.

kfp pipeline create ~/pipelines/samples/core/sequential/<file_name>.yaml -p <pipeline_name>

Verify whether the pipeline has been created using the command:
```
kfp pipeline list
```

Creating a Run¶

To create a run;

Log into the DKubeX homepage.
Click on the workspace’s terminal and create an experiment in the terminal using the command:
```
kfp experiment create <name>
```
Verify whether the experiment has been created using the following command:
```
kfp experiment list
```

Clone the Kubeflow pipelines git repo using the commands:

git clone https://github.com/kubeflow/pipelines.git

cd pipelines/samples/core/sequential/

Convert the .py file into a .yaml to use the file in the pipeline.

kfp dsl compile --py <file_name>.py --output <file_name>.yaml

Create a pipeline using the above .yaml file.

kfp pipeline create ~/pipelines/samples/core/sequential/<file_name>.yaml -p <pipeline_name>

Verify whether the pipeline has been created using the command:
```
kfp pipeline list
```

Create a run in CLI using the following command:

kfp run create -e p1 -r r1 -n sequential -v <pipeline_version>

Note

You can get <pipeline_version> by using the command:

kfp pipeline list-versions <pipeline_id>

Verify whether the run has been created by using the following command:
```
kfp run list
```

Creating a Recurring Run¶

To create a recurring run;

Login to the DKubeX homepage.
Click on the workspace’s terminal and create an experiment in the terminal using the command:
```
kfp experiment create <name>
```
Verify whether the experiment has been created using the following command:
```
kfp experiment list
```

Clone the Kubeflow pipelines git repo using the commands:

git clone https://github.com/kubeflow/pipelines.git

cd pipelines/samples/core/sequential/

Convert the .py file into a .yaml to use the file in the pipeline.

kfp dsl compile --py <file_name>.py --output <file_name>.yaml

Create a pipeline using the above .yaml file.

kfp pipeline create ~/pipelines/samples/core/sequential/<file_name>.yaml -p <pipeline_name>

Verify whether the pipeline has been created using the command:
```
kfp pipeline list
```

Create a recurring run in CLI using the following command:

kfp recurring-run create -enabled --experiment-name <name> --pipeline-id <pipeline_id> --version-id <pipeline_version> --max-concncurrey <concurrent_runs> --job-name <name of run> --start-time <starttime> --end-time <endtime>

kfp recurring-run create -enabled --experiment-name h1--pipeline-id 0e9e4098-f87e-43e1-ac8b-693970508f79 --version-id a6a03f9c-4906-44ba-91e9-a7fc8a16bc60 --max-concurrency 2 --job-name clitesting --start-time 2023-10-17T16:35:00Z --end-time 2023-10-17T16:37:00Z

Verify whether the recurring run has been created using the following command:
```
kfp run list
```

Compare Runs¶

Runs created can be compared through 2 different methods. Either in the CLI or in Kubeflow app in the DKubeX homepage.

Log into the DKubeX homepage.
Click on the workspace’s terminal and create an experiment in the terminal using the command:
```
kfp experiment create <name>
```
Verify whether the experiment has been created using the following command:
```
kfp experiment list
```

Clone the Kubeflow pipelines git repo using the commands:

git clone https://github.com/kubeflow/pipelines.git

cd pipelines/samples/core/sequential/

Convert the .py file to a .yaml to use the file in the pipeline.

kfp dsl compile --py <file_name>.py --output <file_name>.yaml

Create a pipeline using the .yaml file.

kfp pipeline create ~/pipelines/samples/core/sequential/<file_name>.yaml -p <pipeline_name>

Verify whether the pipeline has been created using the command:
```
kfp pipeline list
```

Create a run in CLI using the following command:

kfp run create -e p1 -r r1 -n sequential -v <pipeline_version>

Note

You can get <pipeline_version> by using the command:

kfp pipeline list-versions <pipeline_id>

Verify whether the run has been created using the command:
```
kfp run list
```

Log into the DKubeX homepage.
Click on the workspace’s terminal and create an experiment in the terminal using the command:
```
kfp experiment create <name>
```
Verify whether the experiment has been created using the following command:
```
kfp experiment list
```

Clone the Kubeflow pipelines git repo using the commands:

git clone https://github.com/kubeflow/pipelines.git

cd pipelines/samples/core/sequential/

Go to the File Explorer app in the DKubeX homepage, then go to the directory where you created the <file_name>.yaml and download the file.
After downloading the file, click on the Kubeflow icon in the DKubeX homepage.
Click on Pipelines and click on the + Upload Pipeline option.
Click on Create a new pipeline and click on private and enter any pipeline name and some pipeline description
Upload the <file_name>.yaml that you downloaded from the CLI.

(OR)
Upload <file_name>.yaml by importing the URL.
Click on Create.
Create a run and choose the pipeline, pipeline version and Enter any <run name> and choose the experiment created in the steps above.
Click on One-Off and specify the run parameters required by the pipeline.
Click on Start.
Repeat all of the steps above then go to the Runs page and click on at least 2 runs.
Click on Compare Runs