Use this documentation to deploy the NVIDIA RAG Blueprint Helm chart with NVIDIA MIG (Multi-Instance GPU) slices for fine-grained GPU allocation. For other deployment options, refer to Deployment Options.
To ensure that your GPUs are compatible with MIG, refer to the MIG Supported Hardware List.
Before you deploy, verify that you have the following:
-
A Kubernetes cluster with NVIDIA H100 or RTX PRO 6000 GPUs
:::{note} This section showcases MIG support for
NVIDIA H100 80GB HBM3GPU. The MIG profiles used in themig-config-h100.yamlare specific to this GPU. Refer to the MIG User Guide for MIG profiles of other GPU types. :::
:::{important}
- Ensure that you have at least 200GB of available disk space per node for NIM model caches and application data
- First-time deployment takes 60-70 minutes while large models download without visible progress indicators
For monitoring deployment progress, refer to Deploy on Kubernetes with Helm. :::
-
Verify that you meet the hardware requirements.
-
Verify that you have the NGC CLI available on your client computer. You can download the CLI from https://ngc.nvidia.com/setup/installers/cli.
-
Verify that you have Kubernetes v1.34.2 installed and running on Ubuntu 22.04/24.04. For more information, see Kubernetes documentation and NVIDIA Cloud Native Stack 17.0.
-
Verify that you have installed Helm 3. To install Helm 3 (and avoid Helm 4), follow the official Helm v3 installation instructions for your platform, for example by using the
get-helm-3script described in the Helm documentation. -
Verify that you have a default storage class available in the cluster for PVC provisioning. One option is the local path provisioner by Rancher. Refer to the installation section of the README in the GitHub repository.
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.26/deploy/local-path-storage.yaml kubectl get pods -n local-path-storage kubectl get storageclass
-
If the local path storage class is not set as default, you can make it default by running the following code.
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' -
Verify that you have installed the NVIDIA GPU Operator by using the instructions here.
-
(Optional) You can enable time slicing for sharing GPUs between pods. For details, refer to Time-Slicing GPUs in Kubernetes.
-
Clone the RAG Blueprint Git repository to get access to the MIG configuration files.
-
Verify that you have installed the NVIDIA NIM Operator. If not, install it by running the following code:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ --username='$oauthtoken' \ --password=$NGC_API_KEY helm repo update helm install nim-operator nvidia/k8s-nim-operator -n nim-operator --create-namespace
For more details, see instructions here.
-
Change your directory to deploy/helm/ by running the following code.
cd deploy/helm/ -
Create a namespace for the deployment by running the following code.
kubectl create namespace rag
-
Update the GPU Operator's ClusterPolicy to use the mixed MIG strategy by running the following code.
kubectl patch clusterpolicies.nvidia.com/cluster-policy \ --type='json' \ -p='[{"op":"replace", "path":"/spec/mig/strategy", "value":"mixed"}]'
Edit the MIG configuration file mig-config-h100.yaml to adjust the slicing pattern as needed.
The following example enables a custom configuration with mixed MIG slice sizes on the same GPU.
:::{note} This example uses a custom slicing strategy: 7 slices of 1g.10gb on GPU 0, mixed slices (2x 1g.20gb + 1x 3g.40gb) on GPU 1, and 1 slice of 7g.80gb on GPU 3. This demonstrates the ability to combine different MIG slice sizes on a single GPU for optimal resource utilization. :::
apiVersion: v1
kind: ConfigMap
metadata:
name: custom-mig-config
data:
config.yaml: |
version: v1
mig-configs:
all-disabled:
- devices: all
mig-enabled: false
custom-7x1g10-2x1g20-1x3g40-1x7g80:
- devices: [0]
mig-enabled: true
mig-devices:
"1g.10gb": 7
- devices: [1]
mig-enabled: true
mig-devices:
"1g.20gb": 2
"3g.40gb": 1
- devices: [3]
mig-enabled: true
mig-devices:
"7g.80gb": 1Apply the custom MIG configuration configMap to the node and update the ClusterPolicy, by running the following code.
kubectl apply -n nvidia-gpu-operator -f mig-slicing/mig-config-h100.yaml
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
--type='json' \
-p='[{"op":"replace", "path":"/spec/migManager/config/name", "value":"custom-mig-config"}]'Label the node with MIG configuration, by running the following code.
kubectl label nodes <node-name> nvidia.com/mig.config=custom-7x1g10-2x1g20-1x3g40-1x7g80 --overwrite:::{important} For NVIDIA RTX6000 Pro Deployments:
Use mig-config-rtx6000.yaml instead:
kubectl apply -n nvidia-gpu-operator -f mig-slicing/mig-config-rtx6000.yaml
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
--type='json' \
-p='[{"op":"replace", "path":"/spec/migManager/config/name", "value":"custom-mig-config"}]'
kubectl label nodes <node-name> nvidia.com/mig.config=custom-rtx6000-4x1g24-2x1g24-1x2g48-1x4g96 --overwrite:::
Verify that the MIG configuration is successfully applied, by running the following code.
kubectl get node <node-name> -o=jsonpath='{.metadata.labels}' | jq . | grep migYou should see output similar to the following.
"nvidia.com/mig.config.state": "success"
"nvidia.com/mig-1g.10gb.count": "7"
"nvidia.com/mig-1g.20gb.count": "2"
"nvidia.com/mig-3g.40gb.count": "1"
"nvidia.com/mig-7g.80gb.count": "1"Run the following code to install the RAG Blueprint Helm Chart.
helm upgrade --install rag -n rag https://helm.ngc.nvidia.com/nvidia/blueprint/charts/nvidia-blueprint-rag-v2.5.1.tgz \
--username '$oauthtoken' \
--password "${NGC_API_KEY}" \
--set imagePullSecret.password=$NGC_API_KEY \
--set ngcApiSecret.password=$NGC_API_KEY \
-f mig-slicing/values-mig-h100.yaml:::{important} For NVIDIA RTX6000 Pro Deployments:
If you are deploying on NVIDIA RTX6000 Pro GPUs (instead of H100 GPUs), use values-mig-rtx6000.yaml and mig-config-rtx6000.yaml which include the RTX6000-specific MIG profiles and NIM LLM model configuration.
helm upgrade --install rag -n rag https://helm.ngc.nvidia.com/nvidia/blueprint/charts/nvidia-blueprint-rag-v2.5.1.tgz \
--username '$oauthtoken' \
--password "${NGC_API_KEY}" \
--set imagePullSecret.password=$NGC_API_KEY \
--set ngcApiSecret.password=$NGC_API_KEY \
-f mig-slicing/values-mig-rtx6000.yaml:::
:::{note} Refer to NIM Model Profile Configuration for using non-default NIM LLM profile. :::
:::{note} Due to a known issue with MIG support, currently the ingestion profile has been scaled down while deploying the chart with MIG slicing. This is expected to affect the ingestion performance during bulk ingestion, specifically large bulk ingestion jobs might fail. :::
To view pod GPU assignments, run kubectl-view-allocations as shown following.
kubectl-view-allocationsYou should see output similar to the following.
Resource Requested Limit Allocatable Free
nvidia.com/mig-1g.10gb (86%) 6.0 (86%) 6.0 7.0 1.0
├─ milvus-standalone-... 1.0 1.0
├─ nemotron-embedding-ms-... 1.0 1.0
├─ rag-nv-ingest-... 1.0 1.0
├─ nemoretriever-graphic-elements-v1-... 1.0 1.0
├─ nemoretriever-page-elements-v3-... 1.0 1.0
└─ nemoretriever-table-structure-v1-... 1.0 1.0
nvidia.com/mig-1g.20gb (100%) 2.0 (100%) 2.0 2.0 0.0
├─ nemotron-ranking-ms-... 1.0 1.0
└─ <other-workload> 1.0 1.0
nvidia.com/mig-3g.40gb (100%) 1.0 (100%) 1.0 1.0 0.0
└─ nemoretriever-ocr-v1-... 1.0 1.0
nvidia.com/mig-7g.80gb (100%) 1.0 (100%) 1.0 1.0 0.0
└─ nim-llm-... 1.0 1.0
To check the MIG slices, run the following code from the GPU Operator driver pod.
This runs nvidia-smi within the pod to check GPU MIG slices.
kubectl exec -n gpu-operator -it <driver-daemonset-pod> -- nvidia-smi -LYou should see output similar to the following.
GPU 0: NVIDIA H100 80GB HBM3 (UUID: ...)
MIG 1g.10gb Device 0: ...
MIG 1g.10gb Device 1: ...
MIG 1g.10gb Device 2: ...
MIG 1g.10gb Device 3: ...
MIG 1g.10gb Device 4: ...
MIG 1g.10gb Device 5: ...
MIG 1g.10gb Device 6: ...
GPU 1: NVIDIA H100 80GB HBM3 (UUID: ...)
MIG 1g.20gb Device 0: ...
MIG 1g.20gb Device 1: ...
MIG 3g.40gb Device 2: ...
GPU 3: NVIDIA H100 80GB HBM3 (UUID: ...)
MIG 7g.80gb Device 0: ...
-
Follow the remaining instructions in Deploy on Kubernetes with Helm:
- Ensure you have the correct MIG strategy (
mixed) configured. - Verify that
nvidia.com/mig.config.stateissuccessbefore deploying. - Customize
values-mig-h100.yamlorvalues-mig-rtx6000.yamlto specify the correct MIG GPU resource requests for each pod.