Evaluating LLMs

In this tutorial, we will evaluate the performance of a LLM while comparing it to the performance of OpenAI. For this example, the Llama3.8B model will be used. .. In this tutorial, we will evaluate the performance of a base and finetuned LLM while comparing it to the performance of OpenAI. For this example, the base Llama2-7B and finetuned Llama2-7B models will be used.

Prerequisites

• You need to ingest your data corpus and create a dataset from it. You can refer to the Data ingestion and creating dataset tutorial for more information on how to do this.

  • The dataset name used in this tutorial is contracts.

• You need to deploy the BGE-Large embedding model on DKubeX. To learn how to deploy an embedding model on DKubeX, refer to the Deploying Embedding Models on DKubeX tutorial.

  • The name of the BGE-Large deployment used in this tutorial is bge-large.

• You need to deploy the Llama3-8B LLM model on DKubeX. To learn how to deploy a LLM on DKubeX, refer to the Deploying LLMs in DKubeX tutorial.

  • The names of the Llama3-8B deployment used in this tutorial ia llama38bbase.

• Create a SecureLLM application key with the LLM deployment which will be used in the evaluation process. The steps are provided below.

  • From the DKubeX UI, open and log into the SecureLLM application. Once open, click on the Admin Login button and log in using the admin credentials provided during installation.

    Hint

    In case you do not have the credentials for logging in to SecureLLM, please contact your administrator.

  • On the left sidebar, click on the Keys menu and go to the Application Keys tab on that page.

  • On the API key name field, provide a unique name for the key to be created.

  • From the LLM Keys dropdown list, select DKUBEX.

  • From the Models dropdown list, select your deployed base models (bge-large and llama38base).

  • Click on the Generate Key button.

  • A pop-up window will show up on your screen containing the application key for your new application. Alternatively, you can also access your application key from the list of keys in the Application Key tab. Copy this application key for further use, as it will be required to create the chatbot application. Also make sure that you are copying the entire key including the sk- part.

• Export the following variable to your workspace by running the following commands on your DKubeX Terminal.

  • Replace the <username> part with your DKubeX workspace name.

    export HOMEDIR=/home/<username>
    

• A few .yaml files are required to be used in the evaluation process.

  • On the Terminal application in DKubeX UI, run the following commands:

    wget https://raw.githubusercontent.com/dkubeio/dkubex-examples/refs/tags/v0.8.6.3/rag/evaluation/eval.yaml -P ${HOMEDIR}/
    wget https://raw.githubusercontent.com/dkubeio/dkubex-examples/refs/tags/v0.8.6.3/rag/query/query.yaml -P ${HOMEDIR}/
    

Evaluating LLM Models

To evaluate the Llama3-8B model against OpenAI, follow the steps provided below:

• Provide the appropriate details on the query.yaml file which will be used during the evaluation process. Run vim query.yaml on DKubeX Terminal and provide the following details:

  • On the dataset field, provide the name of the dataset you created earlier, i.e. contracts.

  • On the embedding field, provide the type of the embedding model used for ingestion, i.e. dkubex.

  • In the synthesizer section, provide the following details:

    • On the llm field, make sure dkubex is selected.

    • On the llm_url field, provide the endpoint URL of the deployed model (llama38base) to be used. The endpoint URL can be found on the Deployments page of DKubeX UI.

    • On the llm_key field, provide the serving token for the deployed model (llama38base) to be used. To find the serving token, go to the Deployments page of DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.

  • Under the Embedding Model config section, uncomment the entire dkubex section. Here the details of the embedding model to be used (bge-large) is provided.

    • In the embedding_url field, provide the serving endpoint of the deployment. You can find this by going to the Deployments page in DKubeX UI and clicking on the deployed model name. The serving endpoint will be available on the model details page.

    • In the embedding_key field, provide the serving token for the deployed model to be used. To find the serving token, go to the Deployments page in DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.

  • In the securellm section, provide the following details:

    • On the appkey field, provide the application key that you created earlier on the SecureLLM application.

    • On the dkubex_url field, provide the URL to access DKubeX.

• Provide the appropriate details on the eval.yaml file which will be used during the evaluation process. Run vim eval.yaml and provide the following details:

  • On the dataset field, provide the name of the dataset you created earlier, i.e. contracts.

  • Under the questions_generator section, provide the following details:

    • On the llm field, provide openai.

    • On the llm_url field, keep it blank.

    • On the llm_key field, provide the OpenAI API key.

  • On the rag_configuration field, provide the absolute path to the RAG config (query.yaml) file. In this case it will be /home/<username>/query.yaml, where <username> is the name of your DKubeX workspace.

  • Under the semantic_similarity_evaluator section, provide the following details.

    • On the embedding_provider section, provide dkubex.

    • On the embedding_url field, provide the endpoint URL of the deployed model to be used. The endpoint URL can be found on the Deployments page of DKubeX UI.

    • On the embedding_key field, provide the serving token for the deployed model to be used. To find the serving token, go to the Deployments page of DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.

  • Under the correctness_evaluator section, provide the following details:

    • On the llm field, provide dkubex.

    • On the llm_key field, provide the serving token for the deployed model to be used. To find the serving token, go to the Deployments page of DKubeX UI and click on the deployed model name. The serving token will be available on the model details page.

    • On the llm_url field, provide the endpoint URL of the deployed model to be used. The endpoint URL can be found on the Deployments page of DKubeX UI.

  • On the tracking section, provide a unique name for the MLFlow experiment, allowing for tracking and comparison of different runs of the pipeline.

• Once done, run the following command to start the evaluation process of the Llama3-8B model.

  CommandExample

  d3x dataset evaluate --config ${HOMEDIR}/eval.yaml