Summary

volcano-vgpu-monitor continues exporting pod-level vGPU metrics for completed pods as long as the Pod object still exists in Kubernetes.

This causes stale metrics such as Device_memory_desc_of_container and Device_utilization_desc_of_container to remain visible for pods whose containers have already
terminated.

Environment

• volcano-vgpu-device-plugin: v1.12.0
• Volcano scheduler: v1.13.x
• Kubernetes runtime: containerd
• vGPU mode: HAMi core
• Metrics endpoint: volcano-vgpu-monitor

Problem

After a vGPU pod completes successfully, the Pod phase becomes Succeeded and the container state becomes terminated, but volcano-vgpu-monitor may still export metrics for
that pod.

Example pod:

kubectl get pod task-0

Output:

  NAME   READY   STATUS      RESTARTS   AGE
  task-0   0/1         Completed           0          55m

The pod status shows the container has terminated:

  status:
    phase: Succeeded
    containerStatuses:
    - name: cnt
      state:
        terminated:
          exitCode: 0
          reason: Completed

However, the monitor still keeps the cache directory:
/usr/local/vgpu/vgpu/containers/e8cc7639-da00-43e5-9628-cede669dac2c_cnt/65f331f9-d371-496c-b61e-ede02603e5b3.cache

And volcano-vgpu-monitor logs show it still matches the completed pod:

Pod matched! task-0 test-namespace ...
container matched cnt

As a result, Prometheus can still query pod-level metrics for a completed pod:

Device_memory_desc_of_container{podname="task-0"}
Device_utilization_desc_of_container{podname="task-0"}

Root Cause

The monitor only checks whether the Pod UID still exists in the Kubernetes Pod list.

Current logic:

  func isValidPod(name string, pods *corev1.PodList) bool {
      for _, val := range pods.Items {
          if strings.Contains(name, string(val.UID)) {
              return true
          }
      }
      return false
  }

This means a Succeeded or Failed pod is still considered valid as long as the Pod object has not been deleted.

The check does not verify:

• pod.Status.Phase
• whether the target container is still running
• whether the container has terminated

Expected Behavior

volcano-vgpu-monitor should stop exporting pod-level vGPU metrics after the corresponding container has terminated.

For example:

• Running vGPU pod: metrics should be exported.
• Succeeded pod: metrics should stop being exported.
• Failed pod: metrics should stop being exported.
• Pod object still exists but container is terminated: metrics should stop being exported.

Actual Behavior

Metrics continue to be exported after the pod has completed, until the Pod object is deleted and the cache directory is eventually removed.

Proposed Fix

Instead of only checking whether the Pod UID exists, the monitor should also check whether the corresponding container is still running.

Since the cache directory name contains both Pod UID and container name:

_

the validation can parse both values and check the matching container status.

Suggested behavior:

  func isValidPod(name string, pods *corev1.PodList) bool {
      podUID, containerName := parseContainerDirName(name)

      for _, pod := range pods.Items {
          if string(pod.UID) != podUID {
              continue
          }

          if pod.Status.Phase != corev1.PodRunning {
              return false
          }

          for _, status := range pod.Status.ContainerStatuses {
              if status.Name == containerName && status.State.Running != nil {
                  return true
              }
          }

          return false
      }

      return false
  }

This would prevent stale pod-level metrics from completed or failed containers.

Notes

The current 300-second grace cleanup for invalid pod cache directories can still be preserved. The main change is that completed or failed pods should be treated as invalid for
metric export purposes.

  if !isValidPod(entry.Name(), pods) {
      dirInfo, err := os.Stat(dirName)
      if err == nil && dirInfo.ModTime().Add(time.Second*300).After(time.Now()) {
          continue
      }
      ...
  }

Validation Plan

A patch can be validated with the following cases:

1. Start a vGPU pod and confirm metrics are exported.
2. Let the pod complete successfully while keeping the Pod object.
3. Confirm Device_memory_desc_of_container and Device_utilization_desc_of_container stop being exported for that pod.
4. Run a failed vGPU pod and confirm metrics are also removed.
5. Confirm metrics for still-running vGPU pods are unaffected.