Flyte¶

Flyte empowers users with highly concurrent, scalable, and reproducible workflows tailored for data processing, machine learning, and analytics. It excels in managing, automating, and tracking large-scale machine learning tasks and workloads.

Note

For a broader understanding of the Flyte architecture and usage, please follow this link to read the Flyte User Docs

Note

Flyte exclusively supports EKS clusters and does not currently support RKE clusters.

Note

Flyte provides its users with an example to ease them into its workings. please go to flytesnacks to access this example’s GitHub repository

Prerequisites¶

Before diving into Flyte, ensure you have the necessary components and configurations:

Docker Image: You can either create a Docker image based on your use case or leverage the existing Docker image provided in the examples. Please follow the Building base Docker Image guide for further details.
Workspace Setup: Organize the files required to run your workflow in your workspace. Utilize the FileBrowser application in DKubeX or fetch files directly from a repository using the DKubeX CLI.

For the provided examples, we’ll use the dkubex-examples repository. Clone it and navigate to the Flyte examples directory:

git clone https://github.com/dkubeio/dkubex-examples.git

cd dkubex-examples/flyte-examples

Create a project¶

Once you have a Docker image prepared, run the following command to clone the dkubex-examples repo to your workspace.
```
git clone https://github.com/dkubeio/dkubex-examples.git

cd dkubex-examples/flyte-examples
```

Note

The folder will contain all the necessary files with the default values. If you wish to make any modifications to your flyte project, please go to the folder mentioned above and make the edits accordingly.

Setting Up Flyte¶

Follow these steps to set up Flyte for your project

Create a Flyte Project¶

Use the following command to create a Flyte project. You can customize the YAML file by providing the desired project name:

flytectl create project --file project.yaml

Launch the Flyte Project¶

To initiate the Flyte project, run the following command. Replace <repo> and <tag> with your Docker image details or use the existing repository and tag:

pyflyte run --remote --project flytedemo --image dkubex123/my_flyte_image:latest iris.py optimize_model

Monitoring and Visualization¶

Once your Flyte project is up and running, explore project details conveniently through the Flyte app on the DKubeX homepage.

Project Dashboard¶

The Project Dashboard tab contains the details of all of the workflow & task executions on that domain. You can sort all the executions’ statistics based on their Status, Version, Start Time, Duration, etc.

Workflows¶

The Workflows tab contains the details of the workflows that have been run on that domain. You can view further details of a workflow by clicking on it.

You can sort all the workflows’ statistics based on their Basic information*,Execution **Status, Version, Start Time, Execution Duration, Execution Schedules, etc. Furthermore, you can rerun that particular workflow multiple times through the Launch Workflow button provided on the top right corner of the screen.

Note

You can only relaunch the Workflows already created via the CLI. You cannot create the Workflow(s) in the Flyte App

Tasks¶

The Tasks tab contains the details of the tasks that have been run on that domain. You can view further details of a task by clicking on it.

You can sort all the tasks’ statistics based on their Basic information*, Execution **Status, Version, Start Time, Execution Duration, etc. Furthermore, you can rerun that particular task multiple times through the Launch Task button provided on the top right corner of the screen.

Note

You can only relaunch the Tasks already created via the CLI. You cannot create the Task(s) in the Flyte App

Launch Plans¶

From the Launch plan tab, you can schedule and execute tasks and workflows as per your choice. You can view further details of a launch plan by clicking on it.

You can sort all the Launch plans’ statistics based on their Basic information*, **Status, Versions, Execution Start Time, Execution Duration, etc.