Deploying Embedding Models with SkyPilot

SkyPilot, Deploying Embedding Models on DKubeX

While deploying a model (embedding/LLM) on DKubeX with SkyPilot, first a Sky-serve controller is created locally on Kubernetes. This Sky-serve controller manages all the deployments done on and from this particular DKubeX setup.

You can deploy embedding models on DKubeX with Skypilot in the following ways-

  1. Deploying Embedding Models from Catalog with SkyPilot

Prerequisites

Make sure SkyPilot is configured and set up properly on your DKubeX setup. For details, visit Configuring SkyPilot on DKubeX.

Deploying Embedding Models from Catalog with SkyPilot

  • To check the embedding model catalog on DKubeX, run the following command. This will show the list of all embedding models registered in DKubeX catalog. For the complete list of embedding models registered in DKubeX catalog, visit List of Embedding Models in DKubeX LLM Catalog.

    d3x emb list

  • To deploy an embedding model from the DKubeX catalog, use the following command. Fields to provide proper information are also described below.

    d3x emb deploy --name <deployment-name> --model <model-name> -sky

    • Provide an unique name of the embedding model deployment replacing <deployment-name> in the command.

    • Replace <model-name> with the embedding model name from the DKubeX catalog.

    • Use the --publish flag to make the deployment details available for all users on the DKubeX setup.

    • Use the --min_replicas and --max_replicas flags along with number of replicas to set the minimum and maximum number of replicas configuration for the deployment. For example, --min_replicas 1.

    • If you want to create the deployment with a different type of accelerator from the default one, use the --sky-accelerator flag along with the type and number of accelerators to be used. For example, --sky-accelerator A10G.

  • To check the status of the deployment and the service replica on SkyPilot, run the following command:

    d3x sky status -ra

    Once both the service and the service replica are in ready status, the deployment is ready to use.

    Note

    If the deployment is created with 0 replicas, in that case the deployment service will show a NO REPLICA state. To bring up a service replica for the deployment, send a test request to the service endpoint of the deployment like shown below:

    curl xxx.xxx.xxx.xxx:xxxxx

  • Once the deployment is ready, you can get the service endpoint and the service token of the deployment by visiting the Deployments page in DKubeX and opening the particular deployment’s details page, or by running the following command on the terminal:

    d3x serve list