SecureLLM

Overview

DKubeX SecureLLM is a secure interface solution which facilitates observability and monitoring between AI app users and LLMs in use. For all requests from the AI applications, SecureLLM proxies these LLM calls and records both the request and response transactions.

Features

The main features of SecureLLM are as follows-

1. Proxying all LLM calls, providing better security.

2. Monitoring usage by applications, users and models across interactions with multiple LLMs, helping in managing cost escalation and performance issues.

3. Providing an user-friendly dashboard for monitoring visualization.

4. Distribute custom application keys for each use case with fine grain controls per-app to manage usage and limits.

Install, Upgrade and Uninstall SecureLLM

Note

The following instructions are for installing SecureLLM on a RKE2 cluster. In case of an EKS cluster you can install SecureLLM by using the Terraform scripts during DKubeX installation. For more information regarding this, please visit Installing DKubeX on EKS Cluster using Terraform.

Installing SecureLLM

Prerequisites

You need to get the NFS server ip for your setup for installing SecureLLM.

• On your DKubeX backend terminal, run the following command to get the IP for NFS server of your setup.

  kubectl get pv d3x-data -n d3x -o yaml | grep -i server
  

Installation

• Update the dkubex-helm repository by running this command on your DKubeX backend terminal.

  helm repo update dkubex-helm
  

• Replace the parts enclosed with $ with the appropriate details in the following helm install command and execute it to install securellm. The replacable parts are explained below.

  1. $release-name$: Provide any name

  2. $nfsip$: NFS server IP

  3. $nfspath$: “/securellm/postgres”

     Note

     Using this is safe, but actually you can put anything unique that has not been used before, and especially during DKubeX installation. if unique not used, it deletes whatever was left from before in that path.

  4. $adminid$: Create and put an unique admin ID which will be required to log into SecureLLM. This should be entered in an Email ID format.

  5. $adminpasswd$: Create and put an unique admin password which will be required to log into SecureLLM.

  6. $crtoken$: token for pulling the images for securellm from docker repository.

  7. $openaikey$: You can provide this if you want a default OpenAI key. Otherwise, just keep a single space between quotes.

CommandExample

helm install $release-name$ dkubex-helm/securellm -n securellm --create-namespace \
--set global.db.internal.nfs.enabled=true \
--set global.db.internal.nfs.server="$nfsip$" \
--set global.db.internal.nfs.path=$nfspath$ \
--set sllmAdminUser="$adminid$" \
--set sllmAdminPassword="$adminpasswd$" \
--set ingress.oauth.enabled=true \
--set ingress.className="d3x" \
--set imageCredentials.username="dkubex123" \
--set imageCredentials.password="$crtoken$" \
--set global.db.wipedata=true \
--set sllmOpenaiKey="$openaikey$" \
--version 0.5 --wait

• Wait for this process to be completed. This will install SecureLLM on your setup.

Upgrading SecureLLM

To upgrade SecureLLM from an older version to a newer version, use the following steps.

• Get the name of the deployed release by running the following command.

  helm list -n securellm
  

• Run the following command on your system terminal. Replace the <release-name> with the name of the release you used during installation and the <new version number> with the SecureLLM version you want to upgrade to.

CommandExample

helm upgrade <release-name> dkubex-helm/securellm -n securellm --reuse-values --version <new version number> --wait

Uninstalling SecureLLM

To uninstall SecureLLM, use the following steps.

• Get the name of the deployed release by running the following command.

  helm list -n securellm
  

• Run the following command on your system terminal. Replace the <release-name> with the name of the release you used during installation.

CommandExample

helm uninstall <release-name> -n securellm --wait --cascade=foreground

SecureLLM UI

• From the browser, access the DKubeX UI from the URL provided to you.

  

• On the Apps tab of DKubeX UI, you will find the SecureLLM app icon in the list of user applications provided. Click on the icon.

  

• The SecureLLM Login screen will open with two options: Admin Login and Continue as DKubeX User.

  

Admin Login

The Admin Login gives access to the SecureLLM UI Dashboard and all the features of SecureLLM. To log in as an admin, click on the Admin Login button on the SecureLLM Login screen and provide the admin credentials set during installation.

Note

If you do not have the SecureLLM login credentials, please contact your administrator.

The SecureLLM UI will open. This UI contains all the features of SecureLLM, which are described below.

DashboardRequestsUsersModelsKeysSettings

SecureLLM Dashboard provides graphical representation of data regarding all LLM requests processed via SecureLLM during a specific time range.

• To set the range of time, click on the time filter at the top of the screen. For a custom time range, the exact time and date of the start and end of the range needs to be specified.

  

• A total of 9 sets of data are displayed on this dashboard.

  Costs (USD) per RequestLatencyTokensFeedbackPrompt InjectionJailbreak SimilarityToxicityRefusal SimilarityRequests

  Cost of tokens spent (in USD) plotted against time during the set time range.

  

Continue as DKubeX User

Continuing on SecureLLM as a DKubeX user gives read-only access to the SecureLLM Dashboard and the Requests page. To continue as a DKubeX user, click on the Continue as DKubeX User button on the SecureLLM Login screen. No credentials are required when logging in as a user in SecureLLM.

Using SecureLLM

Managing API Keys

SecureLLM provides the option to add, create and delete API keys. There are two types of API keys in SecureLLM-

1. LLM keys: These are the keys provided by the LLM providers which are used to access the LLMs.

2. Application keys: These are the keys created by users for the applications which are to be onboarded to SecureLLM.

Adding and Deleting OpenAI Keys

Please make sure you have access to a valid OpenAI key. If you do not have access to a valid OpenAI key, please follow the provided steps in the following page: OpenAI API Keys

• Open SecureLLM UI and click on Keys on the left sidebar.

  

• In this page, under the LLM keys tab, the list of all available OpenAI keys will be present. To add the new key, use the following steps.

  • From the LLM Provider dropdown list, select openai.

    

  • In the LLM key name field, provide a unique name for the key to be added.

    

  • In the LLM Key field, provide the newly created OpenAI key and click on the + Add Key button.

    

  • Your newly added OpenAI key will be visible on the list of keys now.

    

• To delete an existing OpenAI key from SecureLLM, apply the following steps-

  • Click on the Delete button at the rightmost end of the row containing the key that you want to delete.

  • Click on the Delete button on the Delete API Key pop-up window. The key will be deleted from your SecureLLM account.

    

Creating and Deleting Application Keys

• Open SecureLLM UI and click on Keys on the left sidebar.

  

• In this page, under the Application keys tab, the list of all available application keys will be present.

• To create a new key for your application, use the following steps:

  • On the API key name field, provide a unique name for the key to be created.

    

  • From the LLM Keys dropdown list, select the LLM key from the LLM Keys list you are going to use with this application key.

    Hint

    If you are going to use models deployed in DKubeX, use the DKUBEX key. Otherwise, use the key of the LLM provider you are going to use.

    

  • From the Models dropdown list, select the LLM model to be used. This list shows the LLM models associated with the LLM key mentioned in the previous step.

    

  • Click on the Generate Key button.

• A pop-up window will show up on your screen containing the application key for your new application. Alternatively, you can also access your application key from the list of keys in the Application Key tab.

  

• To delete an existing application key from SecureLLM,

  • Click on the Delete button at the rightmost end of the row containing the key that you want to delete.

  • Click on the Delete button on the Delete application key pop-up window. The application key will be removed from SecureLLM.

    

Integrating SecureLLM with Application

To integrate SecureLLM with your own application, the following changes should be made in the environment variables of the application.

• Change the host endpoint to <host endpoint url>

• Change the API key of the authorization bearer to the key generated in the Applications page of SecureLLM for that particular application.

For instance, in the following example

curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" \

1. The https://api.openai.com/v1/models part should be replaced with <host endpoint url>

2. The $OPENAI_API_KEY should be replaced with the application key generated and shown in the Applications page.

Monitoring User Requests on SecureLLM

All the requests by all users across all the applications which are onboarded to SecureLLM are recorded. Data regarding all these requests can be monitored by requests, by users or by models.

• Click on the Requests button on the left sidebar on the SecureLLM UI. The Requests page will open, listing all the recorded requests and their data.

  

• Clicking on one of the requests opens a new side-window- Request Information, which contains more detailed information about that particular request.

  

  • On the Metrics tab in the Request Information side window, extensive data and metrics are available.