GPU: make health monitoring optional

[byako]

Could help remediating intel/xpumanager#129 (comment)

@rouke-broersma , @niklasfrick this could help. WDYT ?

[rouke-broersma]

I think it could be interesting to have stats about how many gpus fail to have health stats as well, maybe something for xumpd to expose. Don't personally have a need for it this solution solves my problem from the gpu operator side 👌👌

[eero-t]

LGTM, but naming is marginally misleading because XPUMD provides also GPU capability (mem amount), not just health info.

(In legacy GPU case, they are all iGPUs i.e. have no device memory anyway.)

[niklasfrick]

Nice, this fixes the crashloop on the Gen9.5 boxes from xpumanager#129. With the flag on, the plugin keeps serving allocations while xpumd is uninitialized, which is what I needed. Two things before merge:

1. It still panics on shutdown. waitForXPUMDStream only leaves the loop via err == nil || d.stopXPUMDListener, and returns the last err as-is. In infinite mode against an unavailable xpumd, err is always set, so when the context is cancelled (SIGTERM, drain, rollout) the loop breaks with a non-nil err and the caller hits:

if err != nil {
    panic("xpumd-client: failed to connect to xpumd within expected time, exiting")
}

So every shutdown on an affected node still prints a panic and stacktrace, which is the exact case this PR is meant to avoid. Easiest fix: set err = nil when breaking out because d.stopXPUMDListener is set, or check it (or ctx.Err()) before the panic. Same applies to the reconnect call in the Recv loop.

2. Correct me if I'm wrong but the log probably gets stuck on "attempt 1/30". With attemptStep = 0, attempt never changes, so the log always prints attempt 1/30 no matter how many retries happen. For the case this targets, that's the one line you'd check, and it looks stuck on the first try. A normal loop fixes both this and the step-0 trick:

for attempt := 0; infiniteWait || attempt < ConnectAttemptsMax; attempt++ {
    ...
}

Approach is good and it solves the crashloop. I'd just want (1) fixed before merge; (2) if I am right is an easy win to do at the same time.

[byako]

LGTM, but naming is marginally misleading because XPUMD provides also GPU capability (mem amount), not just health info.

(In legacy GPU case, they are all iGPUs i.e. have no device memory anyway.)

Well, true, but xpumd is not the only source of this info, it's just the default one. GPU DRA driver can query VMEM amount directly from GPU when privileged mode is enabled.

[byako]

You're right, @niklasfrick ! #2 applied, #1 - there's even better way - no need to break out, can return right inside the loop.
Here's log from when xpumd got uninstalled, and soon after, the GPU DRA driver killed:

Details

E0626 11:45:21.850076       1 xpumd.go:107] xpumd-client: error receiving data: EOF
I0626 11:45:21.850141       1 xpumd.go:51] trying to connect to xpumd, attempt 1/30
E0626 11:45:21.851080       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:45:21.954577       1 driver.go:205] UnprepareResourceClaims is called: number of claims: 1
I0626 11:45:21.954589       1 node_state.go:611] Freeing devices from claim c3537aa6-850e-4a78-91ea-ea851d753b96
I0626 11:45:21.954593       1 node_state.go:620] Unpreparing device 0000-00-02-0-0x7d67 (CDI ids: [intel.com/gpu=0000-00-02-0-0x7d67]) for claim c3537aa6-850e-4a78-91ea-ea851d753b96
I0626 11:45:21.954598       1 node_state.go:639] Found allocatable device 0000:00:02.0 for CDI device 0000-00-02-0-0x7d67
I0626 11:45:21.954600       1 node_state.go:620] Unpreparing device 0000-04-00-0-0xe211 (CDI ids: [intel.com/gpu=0000-04-00-0-0xe211 intel.com/gpu-mei=mei1]) for claim c3537aa6-850e-4a78-91ea-ea851d753b96
I0626 11:45:21.954602       1 node_state.go:639] Found allocatable device 0000:04:00.0 for CDI device 0000-04-00-0-0xe211
I0626 11:45:21.954926       1 node_state.go:420] Freeing devices from claim c3537aa6-850e-4a78-91ea-ea851d753b96
I0626 11:45:31.852163       1 xpumd.go:51] trying to connect to xpumd, attempt 2/30
E0626 11:45:31.852322       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:45:36.390037       1 driver.go:171] PrepareResourceClaims is called: number of claims: 0
I0626 11:45:41.853113       1 xpumd.go:51] trying to connect to xpumd, attempt 3/30
E0626 11:45:41.853260       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:45:51.854302       1 xpumd.go:51] trying to connect to xpumd, attempt 4/30
E0626 11:45:51.854356       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:46:01.855268       1 xpumd.go:51] trying to connect to xpumd, attempt 5/30
E0626 11:46:01.855380       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:46:06.391355       1 driver.go:171] PrepareResourceClaims is called: number of claims: 0
I0626 11:46:11.855756       1 xpumd.go:51] trying to connect to xpumd, attempt 6/30
E0626 11:46:11.855891       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:46:21.856151       1 xpumd.go:51] trying to connect to xpumd, attempt 7/30
E0626 11:46:21.856303       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:46:31.856852       1 xpumd.go:51] trying to connect to xpumd, attempt 8/30
E0626 11:46:31.857014       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:46:36.390364       1 driver.go:171] PrepareResourceClaims is called: number of claims: 0
I0626 11:46:41.857694       1 xpumd.go:51] trying to connect to xpumd, attempt 9/30
E0626 11:46:41.857849       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:46:51.858153       1 xpumd.go:51] trying to connect to xpumd, attempt 10/30
E0626 11:46:51.858234       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:47:01.858696       1 xpumd.go:51] trying to connect to xpumd, attempt 11/30
E0626 11:47:01.858793       1 xpumd.go:57] xpumd-client: error calling WatchDeviceHealtherrorrpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory"
I0626 11:47:02.768575       1 helpers.go:156] Received signal 15, exiting.
I0626 11:47:02.768732       1 reflector.go:434] "Stopping reflector" logger="ResourceSlice controller" type="*v1.ResourceSlice" resyncPeriod="0s" reflector="k8s.io/dynamic-resource-allocation/resourceslice/resourceslicecontroller.go:625"
I0626 11:47:02.768732       1 draplugin.go:744] "Stopping"
I0626 11:47:02.768784       1 resourceslicecontroller.go:626] "ResourceSlice informer has stopped" logger="ResourceSlice controller"
I0626 11:47:02.768810       1 resourceslicecontroller.go:229] "Stopping" logger="ResourceSlice controller"
I0626 11:47:02.768896       1 resourceHealthPodStatus.go:56] NodeWatchResources stream closed by client
I0626 11:47:02.768905       1 resourceHealthPodStatus.go:97] Unregistered health stream 1, remaining streams: 0
E0626 11:47:02.768910       1 nonblockinggrpcserver.go:180] "handling stream failed" err="context canceled" logger="dra" requestID=3 method="/v1alpha1.DRAResourceHealth/NodeWatchResources"
I0626 11:47:02.768913       1 nonblockinggrpcserver.go:86] "GRPC server terminated gracefully" logger="dra" endpoint="/var/lib/kubelet/plugins/gpu.intel.com/dra.sock"
I0626 11:47:02.769064       1 nonblockinggrpcserver.go:86] "GRPC server terminated gracefully" logger="registrar" endpoint="/var/lib/kubelet/plugins_registry/gpu.intel.com-reg.sock"
~/git/intel-resource-drivers-for-kubernetes $

Thanks for the review!

[byako]

This is now part of :devel image: https://github.com/intel/intel-resource-drivers-for-kubernetes/pkgs/container/intel-resource-drivers-for-kubernetes%2Fintel-gpu-resource-driver/1014699366?tag=devel