DKube User Guide

Contents:

• Getting Started
  • Overview
  • Initiating Dashboard and Workflow Screens
    • Providing the License Token
    • Authentication Credentials
      • Local Authorization
      • Backend Authorization
      • GitHub Authorization
        • OAuth Details
        • Access Token Details
      • LDAP Authorization
• Using DKube
  • DKube Roles & Workflow
  • First Time Users
  • Data Science Concepts
    • DKube Concepts
    • Pipeline Concepts
    • Projects
      • Projects & Groups
    • Leaderboard
    • FeatureSets
    • Shared Data
    • Tags & Description
    • Delete and Archive
      • Archive
      • Delete
    • Run Scheduling
    • Status Field of IDEs & Runs
  • MLOps Concepts
    • Model Stages
    • Model Serving Transformer
    • Comparing Models
    • Tracking and Lineage
      • Run, Model, and Dataset Lineage
      • Dataset Usage
    • Versioning
  • Container Images
  • Hyperparameter Optimization
    • Katib Within DKube
  • Kubeflow Pipelines
    • Pipeline Definition
    • Pipeline Component
    • Kubeflow Pipelines Within DKube
  • Model Monitoring
  • Multicluster Concepts
    • Multicluster Job Execution
    • Workflow Across Multiple Clusters
      • Deploying on a Different Cluster
  • Operator Concepts
    • Operation of Groups
      • Multi-Group Users
    • Operation of Pools
      • Clustered Pools
    • Default Pool and Group
    • Initial Operator Workflow
• Data Science Tutorial
  • General Workflow
  • Create Code Repo
  • Create Dataset Repo
  • Create Model Repo
  • Create Notebook
    • Basic Submission Screen
    • Repos Submission Screen
  • Open JupyterLab Notebook
  • Create Training Run
    • Basic Submission Screen
    • Repos Submission Screen
      • Input Selections
      • Output Selection
  • View Trained Model
  • Publish Model
  • Deploy Model
• Data Science Dashboard & Workflow
  • Data Science Menu
  • Data Science Dashboard
  • Data Science Workflow
    • Shared Data
  • Multicluster Operation
    • Remote Cluster Execution
    • Multicluster Workflow
      • Deploying on a Different Cluster
  • Projects
    • Project List
    • Create a Project
    • Create and View Project Resources
  • Leaderboard
    • Set up the Leaderboard
    • View the Results
  • Images
    • Add an Image
    • Build an Image
  • Repos
    • Versioned vs Non-Versioned
      • Datasets & Models
      • Program Code
  • Code
    • Add Code
      • Fields for Adding Code
        • Git
    • Delete a Code Repo
  • Datasets
    • Dataset Details
      • Dataset Version Detail
      • Dataset Version Lineage
    • Add a Dataset
      • Datasets Used for Training Input
      • Datasets Used for Preprocessing Output
      • Adding the Dataset
      • Fields for Adding a Dataset
    • Delete a Dataset
  • FeatureSets
    • FeatureSet Sharing
    • Configure the Feature Store
    • Feature Specification
    • Create a FeatureSet
    • Create a new FeatureSet Version
    • Use the FeatureSet in a Training Run
  • Models
    • Model Details
      • Model Version Lineage
    • Add a Model
    • Model Workflow
      • Compare Models
      • Model Actions
      • Edit Model
      • Publish Model
      • Deploy Model
  • IDEs
    • TensorBoard
    • IDE Actions
    • IDE Details
    • Create IDE
      • File Paths for Datasets and Models
      • Basic Submission Screen
        • Code Repo
        • Custom Containers
      • Repo Submission Screen
        • Mount Path for Datasets and Models
      • Configuration Screen
        • Configuration File
        • Hyperparameters
        • GPUs
    • Delete IDE
  • Runs
    • Templates
    • Training Runs
      • TensorBoard
      • Training Run Actions
      • Hyperparameter Optimization
      • Training Run Details
      • Metrics
      • Lineage
      • Logs
      • Compare Runs
    • Create Training Run
      • File Paths for Datasets and Models
      • Basic Submission Screen
        • Code Repo
        • Custom Containers
        • Build From Code Repo
      • Repo Submission Screen
        • Output Model Repo
      • Configuration Submission Screen
        • Target Cluster
        • Configuration File
        • Hyperparameters
        • Hyperparameter Tuning
        • GPU Distribution
      • Stop Run
    • Preprocessing Runs
      • Create Preprocessing Run
      • Basic Submission Screen
      • Repo Submission Screen
  • Kubeflow Pipelines
    • Upload a New Pipeline
    • Create a Pipeline Run
    • Manage Experiments and Runs
    • Sharing Experiments and Runs
    • Archiving and Deleting Experiments & Runs
• Production Engineer
  • Model Deployment Workflow
    • Deploy on the Same Cluster
      • Node Affinity
      • Serving Port & Prefix
      • Deployment Status
      • Change Model Deployment
      • Deployment Details
    • Deploy on a Different Cluster
      • Build and Save the Image on Development Cluster
      • Import the Image on Serving Cluster
• Operator Dashboard & Workflow
  • Operator Menu
  • Operator Dashboard
  • Operator Workflow
  • Pool Management
    • Create Pool
    • Edit Pool
    • Delete Pool
  • Group Management
    • Create Group
    • Edit Group
    • Delete Group
  • User Management
    • Add (On-Board) User
    • Add Users from an LDAP Group
    • Select User Roles
    • Delete User
  • DVS
    • Metadata Source
    • Storage Source
      • S3 or Minio
      • GCS
  • Configuration
    • Feature Store Configuration
  • Cluster Management
    • Cluster Credentials
    • DKube Status Information
    • DKube License Update
    • DKube OAuth Token
  • Multicluster Management
    • Add a Remote Cluster
    • Remote Execution Cluster
    • Remote Monitoring Cluster
      • DKube Remote Cluster
    • Edit a Remote Cluster
    • Remove a Remote Cluster
  • Node & Device Status
• DKube Developer’s Guide
  • File Paths
    • Configuration File
    • Home Directory
  • Amazon S3
    • config.json
    • credentials
    • /etc/dkube/.aws/config
    • /etc/dkube/.aws/credentials
  • Amazon Redshift
    • Redshift Access Configuration
      • Redshift Access with an API Server
      • Redshift Access without an API Server
    • Accessing the Redshift Data from the Program
      • redshift.json
    • Redshift Password
  • Mount Path
  • Environment Variables
    • General Variables
    • Variables Passed to Jobs
    • Repo Variables
    • Redshift Variables
    • Hyperparameter Tuning Variables
  • DKube SDK
  • Writing Code for Metric Logging
    • Import MLFlow Module
    • Create & Set MLFlow Experiment
    • Define Callback Function
    • Train Model & Save Model Data
  • Writing Code for Katib
    • Read the Katib Tuning Input
    • Log the Output Metrics
  • Writing Code for TensorBoard
    • TensorBoard in a Notebook
      • Write the Event Files
    • TensorBoard during a Training Run
      • Write the Event Files
  • Kubeflow Pipelines Template
  • Custom Container Images
    • Manual Image Creation
      • Getting the Base Image
      • Adding Your Packages
      • Building the Docker Image
      • Pushing the Image to Docker Hub
      • Using the Custom Image within DKube
    • JupyterLab Custom Images
  • CI/CD
    • Basic Setup for CI/CD
      • Setting up the Repository
      • Placement of .dkube-ci.yml
    • CI/CD Actions
      • Folder Path
      • Combining Declarations
    • Automated Execution Through GitHub Webhook
    • CI/CD Example
  • Inference Deployment Requirements
    • Optional Transformer
  • Accessing DKube MinIO Server
    • Getting MinIO credentials
    • Creating Dataset Repo
    • TensorFlow Deployment
    • PyTorch Deployment
    • SKLearn Deployment
    • Custom Deployment
• Troubleshooting
  • Resources from Other Users in Your Group are Not Visible
    • Symptom
    • Cause/Action
  • Error Retrieving or Displaying Information
    • Symptom
    • Cause/Action
  • Problems Accessing DKube url
    • Symptom
    • Cause/Action
  • Notebook or Job Loading Slowly
    • Symptom
    • Cause/Action
  • Error Status for Notebook or Run
    • Symptom
    • Cause/Action
  • GPUs are Not Available on Master Node
    • Symptom
    • Cause/Action
    • Cause/Action
  • Jupyter Notebook Takes a Long Time to Load
    • Symptom
    • Cause/Action
  • 404 Error Message from Notebook or Run Screen
    • Symptom
    • Cause/Action
  • Insufficient GPU Message Even Though the Pool is Large Enough
    • Symptom
    • Cause/Action
  • GitHub Users Do Not Appear from On-Board Popup
    • Symptom
    • Cause/Action
  • Data Science View Selection is Not Available after an Upgrade
    • Symptom
    • Cause/Action