Description

External monitoring tools (e.g. Prometheus DCGM, nvidia-smi) periodically show abnormally large GPU memory values for pods using HAMi-core. The reported free memory wraps around to a huge value (e.g. 18 EB) when tracked usage exceeds the configured limit.

Root Cause

1. NVML Hook: Integer Underflow in _nvmlDeviceGetMemoryInfo

File: src/nvml/hook.c

} else {
    switch (version) {
    case 1:
        ((nvmlMemory_t*)memory)->free = (limit-usage);  // BUG: underflow when usage > limit
        ((nvmlMemory_t*)memory)->total = limit;
        ((nvmlMemory_t*)memory)->used = usage;
        return NVML_SUCCESS;
    case 2:
        ((nvmlMemory_v2_t *)memory)->free = (limit-usage);  // BUG: same issue
        ((nvmlMemory_v2_t *)memory)->total = limit;
        ((nvmlMemory_v2_t *)memory)->used = usage;

When usage > limit (due to stale shared memory slots from crashed processes, or race conditions), limit - usage produces a massive size_t value (e.g. 18446744073709551615). External monitoring that calls nvmlDeviceGetMemoryInfo then sees this garbage value.

2. CUDA Hook: Returns Error Instead of Clamping

File: src/cuda/memory.c

} else if (limit < usage) {
    LOG_WARN("limit < usage; usage=%ld, limit=%ld", usage, limit);
    return CUDA_ERROR_INVALID_VALUE;  // BUG: error instead of defensive clamping
}

When this happens inside the CUDA runtime path (cuMemGetInfo_v2), it returns an error code. Downstream callers may not handle this gracefully, leading to inconsistent state.

Why usage > limit Occurs

The shared memory region (/tmp/cudevshr.cache) tracks per-process GPU memory usage. When a process crashes without cleanup, its memory slot is not freed, and the accumulated usage can exceed the limit.

Steps to Reproduce

1. Deploy a pod with HAMi-core GPU memory limit (e.g., volcano.sh/vgpu-memory: 4096)
2. Run GPU workloads that crash or get OOM-killed
3. Monitor via external DCGM exporter or node-level nvidia-smi
4. Observe occasional spikes of free memory showing impossibly large values

Proposed Fix

1. Clamp NVML hook return values (src/nvml/hook.c)

} else {
    // Defensive: clamp usage to limit
    size_t clamped_usage = (usage > limit) ? limit : usage;
    switch (version) {
    case 1:
        ((nvmlMemory_t*)memory)->free = limit - clamped_usage;
        ((nvmlMemory_t*)memory)->total = limit;
        ((nvmlMemory_t*)memory)->used = clamped_usage;
        return NVML_SUCCESS;
    case 2:
        ((nvmlMemory_v2_t *)memory)->free = limit - clamped_usage;
        ((nvmlMemory_v2_t *)memory)->total = limit;
        ((nvmlMemory_v2_t *)memory)->used = clamped_usage;

2. Clamp CUDA hook instead of returning error (src/cuda/memory.c)

} else if (limit < usage) {
    LOG_WARN("limit < usage; usage=%ld, limit=%ld, clamping", usage, limit);
    // Clamp instead of returning error
    CUDA_OVERRIDE_CALL(cuda_library_entry, cuMemGetInfo_v2, free, total);
    size_t actual_limit = (limit > *total) ? *total : limit;
    *free = 0;
    *total = actual_limit;
    return CUDA_SUCCESS;
}

3. Consider periodic stale slot cleanup

Add a mechanism to detect and clean up process slots whose PID is no longer alive, to prevent usage from drifting above limit in the first place.

Impact

• External GPU monitoring (Prometheus/Grafana dashboards) shows unusable data
• Alerting rules may trigger false alarms
• May cause confusion for cluster operators

Environment

• HAMi-core branch: volcano (also affects main)
• Commit: b2bd3dc
• libvgpu.so compiled with -DHOOK_NVML_ENABLE=1 -DHOOK_MEMINFO_ENABLE=1 -DMULTIPROCESS_LIMIT_ENABLE=1