vgpu-monitor panics with invalid UTF-8 when collecting per-container metrics during container initialization

[charford]

Problem

The vgpu-monitor panics on every scrape cycle when a HAMi-managed container is initializing, causing no per-container vGPU metrics to be emitted for any workload.

DeviceUUID() reads the GPU UUID from shared memory written by libvgpu.so inside the container. Before CUDA has fully initialized, this memory is uninitialized/zeroed. The raw bytes are sliced to 40 characters and passed directly as a Prometheus label value to MustNewConstMetric, which panics:

panic: label value "\x80\x00\x00\x00..." is not valid UTF-8

goroutine 426 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
main.ClusterManagerCollector.Collect(...)
    /go/src/volcano.sh/devices/cmd/vgpu-monitor/metrics.go:238

Fix

Validate the UUID is valid UTF-8 before calling MustNewConstMetric. If not, log a warning and skip the device for this scrape cycle. On the next cycle, once libvgpu.so has written the UUID to shared memory, the metric will be collected normally.

Testing

• Verified panic occurs on prod-ltx1-k8s-2 node ltx1-app157276 (pool gpu-nvidia-h100-ssd-hami-kjp) with a PyTorch job requesting only volcano.sh/vgpu-* resources
• Confirmed libvgpu.so and ld.so.preload are correctly injected by the device plugin at container start
• Fix prevents the panic; device will be retried on next scrape once shared memory is initialized

[archlitchi]

/lgtm

[hami-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, charford

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [archlitchi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[archlitchi]

@charford Thanks for this PR, is this issue occurs to HAMi-device-plugin as well?

[charford]

@charford Thanks for this PR, is this issue occurs to HAMi-device-plugin as well?

@archlitchi Yes, the same root cause exists in HAMi-device-plugin. DeviceUUID() is still sliced to 40 chars without UTF-8 validation before being passed to Prometheus as a label value (cmd/vGPUmonitor/metrics.go, collectContainerMetrics).

The impact is less severe though — HAMi uses prometheus.NewConstMetric via a sendMetric wrapper rather than MustNewConstMetric, so it returns an error instead of panicking. The monitor stays up but logs errors and skips per-container metrics for any device whose shared memory hasn't been initialised by libvgpu.so yet.

I've opened a PR with the same fix:

Project-HAMi/HAMi#1703