|
| 1 | +# Deploy with Helm |
| 2 | + |
| 3 | +## Set up Dependencies |
| 4 | +- [Docker](https://docs.docker.com/engine/install/ubuntu/) |
| 5 | +- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) |
| 6 | +- [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) |
| 7 | +- [Minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download) / Kubernetes cluster created using [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) |
| 8 | +- [Helm](https://helm.sh/docs/intro/install/) |
| 9 | + |
| 10 | +## Minikube Specific Configuration |
| 11 | + |
| 12 | +Start Minikube: |
| 13 | +```bash |
| 14 | +minikube start --driver docker --container-runtime docker --gpus all --cpus 8 |
| 15 | +minikube addons enable nvidia-device-plugin |
| 16 | +``` |
| 17 | +Minikube has a limitation when dealing with symbolic links - symbolic links inside a minikube pod can not be created in a mounted path from the host using `minikube mount <host_folder>:<minikube_target_path>`. |
| 18 | + |
| 19 | +Instead, you can copy over the data using `minikube cp <Host models path> /data/nim` command from your host SSD to minikube host. |
| 20 | +in [Values.yaml](generative-virtual-screening-chart/values.yaml), we define the minikube folder path that the PV is created under. |
| 21 | + |
| 22 | +Note, it is important to save the copied files under a [specific locations](https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/) on the minikube container to prevent data loss between reboots. |
| 23 | + |
| 24 | +Copying over a large number of files from your host machine to the minikube container will increase its volume size. You can modify the default path (`/var/lib/docker`) of the docker data dir to be under a dedicated mounted SSD. |
| 25 | +To do so, first stop the docker service |
| 26 | +```bash |
| 27 | +sudo systemctl stop docker |
| 28 | +``` |
| 29 | +Edit (or create) `/etc/docker/daemon.json` and add a `data-root` entry: |
| 30 | +```bash |
| 31 | +{ . |
| 32 | + . |
| 33 | + . |
| 34 | + "data-root": "/path/to/new/docker/data/directory" |
| 35 | +} |
| 36 | +``` |
| 37 | +Copy the existing docker data to: |
| 38 | +```bash |
| 39 | +sudo rsync -avxP /var/lib/docker/ /path/to/new/docker/data/directory |
| 40 | +``` |
| 41 | +Start the docker service |
| 42 | +```bash |
| 43 | +sudo systemctl start docker |
| 44 | +``` |
| 45 | + |
| 46 | +## Configure Cluster & Helm Deployment |
| 47 | +Set your NGC key as a Kubernetes secret: |
| 48 | +```bash |
| 49 | +kubectl create secret generic ngc-registry-secret --from-literal=NGC_REGISTRY_KEY=<YOUR_NGC_REGISTRY_KEY> |
| 50 | +``` |
| 51 | +Set an environment varible with your desired chart name |
| 52 | +```bash |
| 53 | +export CHART_NAME=<your-chart-name> |
| 54 | +``` |
| 55 | +Install Helm Chart: |
| 56 | +```bash |
| 57 | +cd generative-virtual-screening-chart/ |
| 58 | +helm install "${CHART_NAME}" . --debug |
| 59 | +``` |
| 60 | +Uninstall Helm Chart: |
| 61 | +```bash |
| 62 | +cd generative-virtual-screening-chart/ |
| 63 | +helm uninstall "${CHART_NAME}" --wait |
| 64 | +``` |
| 65 | +Test pod GPU Access: |
| 66 | +```bash |
| 67 | +kubectl run gpu-test1 --image=nvidia/cuda:12.6.2-base-ubuntu22.04 --restart=Never --command -- nvidia-smi |
| 68 | +``` |
| 69 | + |
| 70 | +Set up port forwarding to make requests from your local machine to all the 4 services: |
| 71 | +```bash |
| 72 | +kubectl port-forward service/"${CHART_NAME}"-generative-virtual-screening-chart-msa 8081:8081 & \ |
| 73 | +kubectl port-forward service/"${CHART_NAME}"-generative-virtual-screening-chart-openfold2 8082:8082 & \ |
| 74 | +kubectl port-forward service/"${CHART_NAME}"-generative-virtual-screening-chart-genmol 8083:8083 & \ |
| 75 | +kubectl port-forward service/"${CHART_NAME}"-generative-virtual-screening-chart-diffdock 8084:8084 |
| 76 | +``` |
| 77 | + |
| 78 | +## Troubleshooting and Debugging Kubernetes Pods |
| 79 | +List all pods: |
| 80 | +```bash |
| 81 | +kubectl get pods |
| 82 | +``` |
| 83 | + |
| 84 | +Display detailed information about a specific pod: |
| 85 | +```bash |
| 86 | +kubectl describe pod <pod_name> |
| 87 | +``` |
| 88 | + |
| 89 | +View the logs of a pod: |
| 90 | +```bash |
| 91 | +kubectl logs <pod_name> |
| 92 | +``` |
| 93 | + |
| 94 | +Open an interactive shell in a pod's container: |
| 95 | +```bash |
| 96 | +kubectl exec -it <pod name> -- bash |
| 97 | +``` |
| 98 | + |
| 99 | +## Running the Blueprint With the Helm Deployment |
| 100 | + |
| 101 | +Note, due to the large size of the model files, a substantial `initialDelay` was set for the `livenessProbe` and `readinessProbe` in [each deployment](./templates/). This extended delay prevents premature pod termination, allowing sufficient time for the model files to load before the probes begin their checks. |
| 102 | +You can adjust these values once the models are downloaded and cached in the PV. |
| 103 | + |
| 104 | +Before executing the notebook, ensure each pod's web server is actively listening and ready to handle incoming requests. Verify this by checking the [pod log output](#troubleshooting-and-debugging-kubernetes-pods). If you attempt to access a pod before it is fully operational, the port-forwarding command will terminate. In such case, you’ll need to execute `pkill -f "kubectl port-forward"` to stop any existing port-forwarding processes, then reinitiate the port-forwarding command as described [here](#configure-cluster--helm-deployment) |
| 105 | + |
| 106 | +Run the [protein-binder-design.ipynb](../src/protein-binder-design.ipynb) notebook |
0 commit comments