Deploying LLMs in DKubeX¶
Both base and finetuned LLMs can be deployed in DKubeX. The steps to deploy them are given below.
Note
To make the LLM deployment accessible for all users on a particular DKubeX setup, please use the --public
flag in the deployment command.
Deploying Base LLMs¶
You can deploy base LLMs which are registered with the DKubeX LLM Registry and base LLMs available on the Huggingface repository.
To list all base LLMs registered with DKubeX, use the following command.
d3x llms list
Information
To see the full list of LLMs registered with DKubeX LLM Registry, please visit the List of LLMs in DKubeX LLM Catalog page.
To deploy a base LLM registered with the DKubeX LLM registry, use the following command. Replace the parts enclosed within <> with the appropriate details.
Note
In case you are using a EKS setup, please change the value of the flag --type from a10 to g5.4xlarge in the following command.
d3x llms deploy --name <name of the deployment> --model <LLM Name> --type <GPU Type> --token <access token for the model (if required)>
d3x llms deploy --name llama38b --model meta-llama/Meta-Llama-3-8B-Instruct --type a10 --token hf_Ahq***********jWmO
You can check the status of the deployment from the Deployment page in DKubeX or by running the following command.
d3x serve list
You can deploy base LLMs not available in the DKubeX LLM Registry from the Huggingface repository. Replace the parts enclosed within <> with the appropriate details.
Attention
Make sure you have provided the deployment configuration file for the LLM that you want to deploy in your workspace.
Note
In case you are using a EKS setup, please change the value of the flag --type from a10 to g5.4xlarge in the following command.
d3x llms deploy --name <deployment name> --config <path to deployment config file> --type <GPU type> --token <access token for the model (if required)>
d3x llms deploy --name llama38b --config /home/demo/llama38b.yaml --type a10 --token hf_Ahq***************WmO
You can check the status of the deployment from the Deployment page in DKubeX or by running the following command.
d3x serve list
Deploying Finetuned LLMs¶
You can deploy LLMs finetuned and saved in your workspace, or you can also deploy finetuned LLMs registered in MLFlow in your workspace.
To deploy a finetuned LLM saved in your workspace, use the following command. Replace the parts enclosed within <> with the appropriate details.
Note
In case you are using a EKS setup, please change the value of the flag --type from a10 to g5.4xlarge in the following command.
d3x llms deploy -n <name of the deployment> --base_model <base LLM name> -m <absolute path to the finetuned model> --type <GPU type> --token <access token for the model (if required)>
d3x llms deploy -n llama38bft --base_model meta-llama/Meta-Llama-3-8B-Instruct -m /home/demo/finetuned_llama38b --type a10 --token hf_Ahq*********************WmO
You can check the status of the deployment from the Deployment page in DKubeX or by running the following command.
d3x serve list
To deploy a finetuned LLM registered in MLFlow in your workspace, use the steps provided below:
To list all LLMs registered in MLFlow, use the following command.
d3x models list
To deploy a finetuned LLM registered in MLFlow, use the following command. Replace the parts enclosed within <> with the appropriate details.
Note
In case you are using a EKS setup, please change the value of the flag --type from a10 to g5.4xlarge in the following command.
d3x llms deploy -n <name of the deployment> --base_model <base LLM name> --mlflow <name of registered model>:<model version> --type <GPU type> --token <access token for the model (if required)>
d3x llms deploy -n llama38bft --base_model meta-llama/Meta-Llama-3-8B-Instruct --mlflow llama38b:1 --type a10 --token hf_Ahq****************WmO
You can check the status of the deployment from the Deployment page in DKubeX or by running the following command.
d3x serve list