List of LLMs in DKubeX LLM Catalog

The following list contains the base LLMs which are currently registered with the DKubeX LLM Catalog on the current version of DKubeX (DKubeX v0.8.5.3). These LLMs can be deployed on the DKubeX platform without using any config files provided the resource and permission requirements are met. You can deploy these LLMs on DKubeX using local resources or using cloud resources using SkyPilot.

LLMs registered on the DKubeX LLM Registry

LLM Name	Required GPU Type	GPU Memory Utilization	Max number of batched tokens	Max number of sequences	Trust remote code	Max number of total tokens	Number of GPUs needed
casperhansen/llama-3-70b-instruct-awq	a10	0.85	4096	64	true	4096	1
meta-llama/Meta-Llama-3-8B-Instruct	a10	0.85	4096	64	true	4096	1
meta-llama/Meta-Llama-3.1-8B-Instruct	a10	0.85	4096	64	true	4096	1
microsoft/Phi-3-small-8k-instruct	a10	0.85	4096	64	true	2048	1
mistralai/Mistral-7B-Instruct-v0.2	a10	0.7	4096	64	true	4096	1

These LLMs can be deployed on the DKubeX platform without using any config files provided the resource and permission requirements are met. You can deploy these LLMs on DKubeX using local resources or using cloud resources using SkyPilot.

Note

For more information regarding deploying LLMs on DKubeX, refer to Deploying LLMs in DKubeX.

Deploying LLMs using local resourcesDeploying LLMs using SkyPilot

To deploy an LLM using local resources, you can use the command given below. Replace the following fields in the command with the appropriate values:

• <name of the deployment>: Unique name for the LLM deployment

• <LLM Name>: Name of the LLM provided in the list above

• <GPU Type>: GPU type required for the LLM

• <access token>: Access token required for the LLM (if any)

Note

• In case you are using a EKS setup, please change the value of the flag --type from a10 to g5.4xlarge in the following command.

• Use the --publish flag if you want to make the deployment available for any user on the same setup to access and use.

CommandExample

d3x llms deploy --name <name of the deployment> --model <LLM Name> --type <GPU Type> --token <access token> --publish