Quickstart¶

In this example we will explore the basic features of DKubeX. We will start by deploying an embedding model and a LLM model, ingesting a dataset from our workspace, and building a RAG chat application. The steps are as follows:

Ingesting Data
Deploying embedding models and LLMs from Model Catalog
Building your first RAG chat application

Prerequisites¶

You must have the current version of DKubeX installed into your system. For detailed instructions regarding installation and logging in to DKubeX, please refer to Installation and Logging In.
For this example, you need a A10 GPU (g5.4xlarge) node attached to your DKubeX cluster. In case of a RKE2 cluster, make sure the node is labeled as a10 during installation.
Make sure you have the Huggingface access token to deploy the models.
Export the following variable by running the following commands on your DKubeX Terminal. Replace the <username> part with your DKubeX username.
```
export HOMEDIR=/home/<username>
```
This example uses a small subset of the ContractNLI dataset throughout where required. You need to download this dataset subset on your DKubeX workspace.

Attention

Although the ContractsNLI dataset is in accordance with the terms and conditions of the Creative Commons Attribution 4.0 International Public License, it is recommended to go through the terms and conditions of the dataset before using it. You can read the terms and conditions here- https://stanfordnlp.github.io/contract-nli/#download
- To download the dataset and extract the documents, open the Terminal application from the DKubeX UI and run the following commands:
```
wget https://raw.githubusercontent.com/dkubeio/dkubex-examples/refs/tags/v0.8.7.1/rag/sample-datasets/contract-nli.zip -P ${HOMEDIR}/
unzip ${HOMEDIR}/contract-nli.zip && rm -rf ${HOMEDIR}/contract-nli.zip
```

Deploying embedding models and LLMs from Model Catalog¶

Note

For detailed information regarding this section, please refer to Deploying LLMs in DKubeX.

Here we will deploy the BGE-Large embedding model and Llama3-8B LLM, which is already pre-registered with DKubeX.

Note

This workflow requires an a10 GPU node. Make sure your cluster is equipped with such. Also, in case you are using any other type of GPU node, make sure to use the label for that node which you have put during DKubeX installation process.

Export the Huggingface access token for the models. Replace the <Huggingface token for Llama3-8B> part with the the access token.
```
export HF_TOKEN="<Huggingface token for Llama3-8B>"
```
To list all embedding models registered with DKubeX, use the following command.
```
d3x emb list
```

To deploy the BGE-Large embedding model, use the following command.

d3x emb deploy --name=bge-large --model=BAAI--bge-large-en-v1-5 --token ${HF_TOKEN} --kserve

To list all LLM models registered with DKubeX, use the following command.
```
d3x llms list
```
Deploy the base Llama3-8B model using the following command.
```
d3x llms deploy --name=llama38bbase --model=meta-llama/Meta-Llama-3-8B-Instruct --token ${HF_TOKEN} --type=a10 --kserve
```
Note

In case you are using a EKS setup, please change the value of the flag --type from a10 to g5.4xlarge in the following command. Also, in case you are using any other type of GPU node, make sure to use the label for that node which you have put during DKubeX installation process.
You can check the status of the deployments from the Deployments page in DKubeX or by running the following command.
```
d3x serve list
```
Wait until the deployments are in running state.

Ingesting Data¶

Note

For detailed information regarding this section, please refer to Data ingestion and creating dataset.

Important

This example uses the BAAI/bge-large-en-v1.5 embeddings model for data ingestion.

Configuring `.yaml` file for ingestion¶

You need to provide a configuration .yaml file to be used during the ingestion process.

On the Terminal application in DKubeX UI, run the following command to pull the ingestion configuration file.

wget https://raw.githubusercontent.com/dkubeio/dkubex-examples/refs/tags/v0.8.7.1/rag/ingestion/ingest.yaml -P ${HOMEDIR}/

You need to provide proper details on the ingest.yaml file. Run vim ${HOMEDIR}/ingest.yaml and make the following changes.
- On the embedding section, select dkubex as we are going to use the BAAI/bge-large-en-v1.5 embedding model deployment we did earlier.
- On the reader section, select file as we are going to use the file reader from Llamaindex to read the documents for ingestion. For more information regarding the file reader, visit the Llamaindex documentation.
- Uncomment the entire dkubex section under Embedding Model Details. Here the details of the embedding model to be used (bge-large) is provided. Provide the following details:
  - In the embedding_url field, provide the serving endpoint of the embedding deployment. You can find this by going to the Deployments page in DKubeX UI and clicking on the deployed model name. The serving endpoint will be available on the model details page.
  - In the embedding_key field, provide the serving token for the deployed model to be used. To find the serving token, go to the Deployments page in DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.
- Make sure the file section under Data Reader Details is uncommented. Under here, in the input_dir field provide the absolute path to your dataset folder, i.e. in this case, /home/<your username>/contract-nli/ (Provide your DKubeX username in place of <your username>).
You can also modify and customize several other options in the ingest.yaml file according to your needs, including the splitter class, chunk size, embedding model to be used, etc.

Triggering ingestion and creating dataset¶

Open the Terminal application in DKubeX UI.
Use the following command to perform data ingestion and create the dataset. A dataset named contracts will be created.
```
d3x dataset ingest -d contracts --config ${HOMEDIR}/ingest.yaml --faq
```
Note
- A few documents from the ContractsNLI dataset may show errors during the ingestion process. This is expected behaviour ar those documents’ format are not suitable for ingestion.
- The time taken for the ingestion process to complete depends on the size of the dataset. The ContractsNLI dataset contains 605 documents and the ingestion process may take around 30 minutes to complete. Please wait patiently for the process to complete.
- In case the terminal shows a timed-out error, that means the ingestion is still in progress, and you will need to run the command provided on the CLI after the error message to continue to get the ingestion logs.
- The record of the ingestion and related artifacts are also stored in the MLFlow application on DKubeX UI.
To check if the dataset has been created, stored and are ready to use, use the following command:
```
d3x dataset list
```
To check the list of documents that has been ingested in the dataset, use the following command:
```
d3x dataset show -d contracts
```

Building your first RAG chat application¶

Note

For detailed information regarding this section, please refer to Creating and accessing the chatbot application.

From the DKubeX UI, go to the Terminal application.
You will need to configure and use the query.yaml file from the dkubex-examples repo to be used in the query process in the Securechat application.
- Run the following command to put the query.yaml file on your workspace.
```
wget https://raw.githubusercontent.com/dkubeio/dkubex-examples/refs/tags/v0.8.7.1/rag/query/query-quickstart.yaml -P ${HOMEDIR}/
```
- Run vim ${HOMEDIR}/query-quickstart.yaml and provide the following details on the query-quickstart.yaml file. Once provided, save the file.
  - On the dataset field, provide the name of the dataset you created earlier, i.e. contracts.
  - On the embedding field, provide the type of the embedding model used for ingestion, i.e. dkubex.
  - In the synthesizer section, provide the following details:
    - On the llm field, make sure dkubex is selected.
    - On the llm_url field, provide the endpoint URL of the deployed model (llama38base) to be used. The endpoint URL can be found on the Deployments page of DKubeX UI.
    - On the llm_key field, provide the serving token for the deployed model (llama38base) to be used. To find the serving token, go to the Deployments page of DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.
  - Under the Embedding Model config section, provide the following details:
    - On the embedding_url field, provide the endpoint URL of the deployed model to be used. To find the endpoint URL, go to the Deployments page of DKubeX UI and click on the deployed model name. The endpoint URL will be available on the model details page.
    - On the embedding_key field, provide the serving token for the deployed model to be used. To find the serving token, go to the Deployments page of DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.
You will need to configure and use the securechat.yaml file from the dkubex-examples repo to configure and create the chatbot application.
- Run the following command to put the securechat.yaml file on your workspace.
```
wget https://raw.githubusercontent.com/dkubeio/dkubex-examples/refs/tags/v0.8.7.1/rag/securechat/securechat.yaml -P ${HOMEDIR}/
```
- Run vim ${HOMEDIR}/securechat.yaml and provide the following details on the securechat.yaml file. Once provided, save the file.
  - On the name: field, provide a unique name to be used for the chatbot application. In this example, we will use the name ndabase.
  - On the env:SECUREAPP_ACCESS_KEY: field, provide a password which will be used to access the chatbot application.
  - On the env:FMQUERY_ARGS: field, provide the following details:
    - Use the argument llm to specify that the chatbot application will use the LLM model deployment (llama38bbase) in DKubeX.
    - Provide the name of the dataset, i.e. contracts after the --dataset flag.
    - Provide the absolute path of the query-quickstart.yaml file that you created earlier after the --config flag. It should be /home/<your username>/query-quickstart.yaml for this example. Replace the <your username> part with your DKubeX username.
  - On the ingressprefix: field, provide a unique prefix to be used for the chatbot application. In this example, we will use the prefix /ndabase. This will be used in the application URL. e.g. https://123.45.67.89:32443/ndabase

Launch the app deployment with the following command:

d3x apps create --config ${HOMEDIR}/securechat.yaml

To check the status of the app deployment, use the following command:
```
d3x apps list
```
Once the app deployment status becomes running, you can access the application from the Apps page of DKubeX UI. Provide the application key that you set in the SECUREAPP_ACCESS_KEY field earlier to start using the chat application.

Hint

You can ask the following questions to the chatbot when using the ContractsNLI dataset:

How do I frame a confidential information clause?
What is the difference between a unilateral and mutual NDA?
What are some common exceptions to confidential information clauses?

Tutorials and More Information¶

For more examples including how to train and register models and deploy user applications, please visit the following pages and go through the table provided:

Training Example

Training Fashion MNIST model in DKubeX
Finetuning open-source LLMs

./training/index.html

Serving Example

Deploying models registered in MLFlow in DKubeX
Deploying models from Huggingface repo
Deploying LLM registered in DKubeX
Deploying embedding models registered in DKubeX

./serving/index.html

RAG

Creating a Securechat App using BGE-large Embeddings and Llama3-8b Summarisation Models

./rag/rag.html

SkyPilot

Data ingestion with SkyPilot
Embedding model deployment with SkyPilot
LLM deployment with SkyPilot
Wine Model Finetuning using Skypilot
Llama2 Finetuning using SkyPilot

./skypilot/skypilot.html

Quickstart¶

Prerequisites¶

Deploying embedding models and LLMs from Model Catalog¶

Ingesting Data¶

Configuring .yaml file for ingestion¶

Triggering ingestion and creating dataset¶

Building your first RAG chat application¶

Tutorials and More Information¶

Configuring `.yaml` file for ingestion¶