make kubelet plugin dir configurable

[guptaNswati]

Backport kubernetes-sigs/dra-example-driver#98 to make kubelet dir path configurable via helm.

fixes: #339

Follow-ups:

• Add liveness probe to kubelet-plugin  #453
• fix: support seamless upgrades with pod uid in socket path names + default helm chart values dra-example-driver#113

Changes tested in local kind cluster.

$ git diff
helm upgrade -i --create-namespace --namespace nvidia-dra-driver-gpu nvidia-dra-driver-gpu ${PROJECT_DIR}/deployments/helm/nvidia-dra-driver-gpu \
     ${NVIDIA_DRIVER_ROOT:+--set nvidiaDriverRoot=${NVIDIA_DRIVER_ROOT}} \
     ${MASK_NVIDIA_DRIVER_PARAMS:+--set maskNvidiaDriverParams=${MASK_NVIDIA_DRIVER_PARAMS}} \
+    --set kubeletPlugin.kubeletRegistrarDirectoryPath=/var/lib/kubelet2/plugins_registry \
+    --set kubeletPlugin.kubeletPluginsDirectoryPath=/var/lib/kubelet2/plugins \

@@ -44,6 +44,7 @@ nodes:
     nodeRegistration:
       kubeletExtraArgs:
         v: "1"
+        root-dir: /var/lib/kubelet2
 - role: worker
   labels:
     node-role.x-k8s.io/worker: ""
@@ -53,6 +54,7 @@ nodes:
     nodeRegistration:
       kubeletExtraArgs:
         v: "1"
+        root-dir: /var/lib/kubelet2

kubectl get po -n nvidia-dra-driver-gpu 
NAME                                                READY   STATUS    RESTARTS   AGE
nvidia-dra-driver-gpu-controller-58b49ff77f-zf4f5   1/1     Running   0          6m10s
nvidia-dra-driver-gpu-kubelet-plugin-v67jv          2/2     Running   0          6m10s

nonblockinggrpcserver.go:90] "GRPC server started" logger="dra" endpoint="/var/lib/kubelet2/plugins/compute-domain.nvidia.com/dra.sock"
I0815 19:30:49.872525       1 nonblockinggrpcserver.go:90] "GRPC server started" logger="registrar" endpoint="/var/lib/kubelet2/plugins_registry/compute-domain.nvidia.com-reg.sock"

I0815 19:30:50.068397       1 nonblockinggrpcserver.go:90] "GRPC server started" logger="dra" endpoint="/var/lib/kubelet2/plugins/gpu.nvidia.com/dra.sock"
I0815 19:30:50.068501       1 nonblockinggrpcserver.go:90] "GRPC server started" logger="registrar" endpoint="/var/lib/kubelet2/plugins_registry/gpu.nvidia.com-reg.sock"

kubectl logs pod -n gpu-test2 
Defaulted container "ctr0" out of: ctr0, ctr1
GPU 0: NVIDIA GB200 (UUID: GPU-dcf75458-101f-9395-1d4c-7c78d15bd3c2)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[guptaNswati]

The driver is still trying to register at old path. Need to debug more...

+++ b/demo/clusters/kind/install-dra-driver-gpu.sh
@@ -27,6 +27,8 @@ kubectl label node -l node-role.x-k8s.io/worker --overwrite nvidia.com/gpu.prese
 helm upgrade -i --create-namespace --namespace nvidia-dra-driver-gpu nvidia-dra-driver-gpu ${PROJECT_DIR}/deployments/helm/nvidia-dra-driver-gpu \
     ${NVIDIA_DRIVER_ROOT:+--set nvidiaDriverRoot=${NVIDIA_DRIVER_ROOT}} \
     ${MASK_NVIDIA_DRIVER_PARAMS:+--set maskNvidiaDriverParams=${MASK_NVIDIA_DRIVER_PARAMS}} \
+    --set kubeletPlugin.kubeletRegistrarDirectoryPath=/var/lib/kubelet2/plugins_registry \
+    --set kubeletPlugin.kubeletPluginsDirectoryPath=/var/lib/kubelet2/plugins \

--- a/demo/clusters/kind/scripts/kind-cluster-config.yaml
+++ b/demo/clusters/kind/scripts/kind-cluster-config.yaml
@@ -44,6 +44,7 @@ nodes:
     nodeRegistration:
       kubeletExtraArgs:
         v: "1"
+        root-dir: /var/lib/kubelet2
 - role: worker
   labels:
     node-role.x-k8s.io/worker: ""
@@ -53,6 +54,7 @@ nodes:
     nodeRegistration:
       kubeletExtraArgs:
         v: "1"
+        root-dir: /var/lib/kubelet2

$ ps aux | grep kubelet | grep -v grep | grep root-dir
root     3872652  5.3  0.0 8109376 36864 ?       Ssl  21:47   0:15 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=172.18.0.2 --node-labels=node-role.x-k8s.io/control-plane= --pod-infra-container-image=registry.k8s.io/pause:3.10 --provider-id=kind://docker/k8s-dra-driver-gpu-cluster/k8s-dra-driver-gpu-cluster-control-plane --root-dir=/var/lib/kubelet2 --v=1 --runtime-cgroups=/system.slice/containerd.service

root@k8s-dra-driver-gpu-cluster-control-plane:/# cat /var/lib/kubelet/kubeadm-flags.env 
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=172.18.0.2 --node-labels=node-role.x-k8s.io/control-plane= --pod-infra-container-image=registry.k8s.io/pause:3.10 --provider-id=kind://docker/k8s-dra-driver-gpu-cluster/k8s-dra-driver-gpu-cluster-control-plane --root-dir=/var/lib/kubelet2 --v=1"

$ kubectl get po nvidia-dra-driver-gpu-kubelet-plugin-gtznm  -n nvidia-dra-driver-gpu  -o yaml | grep /var/lib
    - mountPath: /var/lib/kubelet2/plugins_registry
    - mountPath: /var/lib/kubelet2/plugins
    - mountPath: /var/lib/kubelet2/plugins_registry
    - mountPath: /var/lib/kubelet2/plugins
      path: /var/lib/kubelet2/plugins_registry
      path: /var/lib/kubelet2/plugins
    - mountPath: /var/lib/kubelet2/plugins_registry
    - mountPath: /var/lib/kubelet2/plugins
    - mountPath: /var/lib/kubelet2/plugins_registry
    - mountPath: /var/lib/kubelet2/plugins

Error: error creating driver: start registrar: start gRPC server: listen on "/var/lib/kubelet/plugins_registry/compute-domain.nvidia.com-reg.sock": listen unix /var/lib/kubelet/plugins_registry/compute-domain.nvidia.com-reg.sock: bind: no such file or directory

its coming from ./vendor/k8s.io/dynamic-resource-allocation/kubeletplugin/draplugin.go which has a hard-coded path 

[nojnhuh]

I see a few things missing that are causing that error. These changes should get things further along: guptaNswati/k8s-dra-driver@kubelet-dir-flags...nojnhuh:k8s-dra-driver:pr/464

[guptaNswati]

Oh i missed the envs but why do we need the kubelet path in for compute-domain-daemon? And atleast this should not cause the socket registation error

[nojnhuh]

Sorry I got turned around amongst the compute-domain binaries. I see you already have the flags on the kubelet-plugin binary which is where I meant to add that.

[guptaNswati]

Adding envs worked. Thank you @nojnhuh :)

[klueska]

Just a few small comments, but looking good overall.

[jgehrcke]

Oh. I noted that in another PR: this is not where the kubelet stores plugin registrations. This is the directory where the kubelet looks for unix domain sockets to discover plugins. A plugin registration then is performed through such socket.

I'd rather have no help text than a misleading one. Feel free to act on this opinion as you wish (I am OK with a merge).

[guptaNswati]

Yes. see this is how upstream describes it https://github.com/kubernetes/dynamic-resource-allocation/blob/master/kubeletplugin/draplugin.go#L226

go doc --all k8s.io/dynamic-resource-allocation/kubeletplugin.RegistrarDirectoryPath
package kubeletplugin // import "k8s.io/dynamic-resource-allocation/kubeletplugin"

func RegistrarDirectoryPath(path string) Option
    RegistrarDirectoryPath sets the path to the directory where the kubelet
    expects to find registration sockets of plugins. Typically this is
    /var/lib/kubelet/plugins_registry with /var/lib/kubelet being the kubelet's
    data directory.

    This is also the default. Some Kubernetes clusters may use a different data
    directory. This path must be the same inside and outside of the driver's
    container. The directory must exist.

i am okay either way. i would prefer not to deviate from upstream as a different description may confuse more.

[klueska]

Let's leave it as is for now. Feel free to merge.