[Initiative]: GPU-Aware Autoscaling in Cloud Native AI Infrastructure

[pmady]

Name: GPU-Aware Autoscaling in Cloud Native AI Infrastructure

Short description: Whitepaper on GPU autoscaling in Kubernetes using KEDA external scaler with direct NVML metrics

Responsible group: TOC

Does the initiative belong to a subproject: No

Primary contact: @pmady (Pavan Madduri)

Additional contacts: @JulioPerez (Julio Perez - AI TCG Organizer)

Initiative description:
Kubernetes HPA can't see GPU utilization — it watches CPU/memory while GPUs sit at 100%. The current fix (DCGM exporter → Prometheus → KEDA) adds 15-30s latency and a lot of moving parts.

This whitepaper documents a direct NVML approach using KEDA's external scaler pattern:

• Architecture constraints: Why GPU support can't go in KEDA core (CGO, node-local device access)
• DaemonSet + gRPC design: Direct NVML reads with sub-second latency
• Scaling profiles: Pre-built configs for vLLM, Triton, training, batch workloads
• NUMA integration: Works with Volcano's GPU NUMA-aware scheduling
• Production data: 4-node A100 cluster running LLM inference
• Ecosystem fit: Complements DCGM, HAMi, KubeAI (doesn't replace them)

Draft status: Complete whitepaper ready for review (12 sections, 274 lines)
Code implementation: https://github.com/pmady/keda-gpu-scaler (in production)

Deliverable(s) or exit criteria:

• Whitepaper reviewed and approved by TOC
• Published on CNCF TAG Infrastructure website
• Presented at CNCF KubeCon or TAG session
• Reference implementation (keda-gpu-scaler) shows adoption

[pmady]

PS: I also wrote a CNCF Blog post about this work that was published in May: https://www.cncf.io/blog/2026/05/27/gpu-autoscaling-on-kubernetes-with-keda-building-an-external-scaler/

The blog is more high-level/consumer-focused, while this whitepaper goes deep on architecture constraints, NVML integration, and production observations. They serve different audiences.