Add vfio-gpu profile for testing KubeVirt DRA

.gitignore

@@ -1,4 +1,5 @@
 .cache/
+.config/
 vendor/
 [._]*.sw[a-p]
 

README.md

@@ -25,7 +25,7 @@ The procedure below has been tested and verified on both Linux and Mac.
 * [GNU Make 3.81+](https://www.gnu.org/software/make/)
 * [GNU Tar 1.34+](https://www.gnu.org/software/tar/)
 * [docker v20.10+ (including buildx)](https://docs.docker.com/engine/install/) or [Podman v4.9+](https://podman.io/docs/installation)
-* [kind v0.17.0+](https://kind.sigs.k8s.io/docs/user/quick-start/)
+* [kind v0.32.0+](https://kind.sigs.k8s.io/docs/user/quick-start/) (required for Kubernetes 1.36 node images / containerd config v4; `kind load` fails on older kind)
 * [helm v3.7.0+](https://helm.sh/docs/intro/install/)
 * [kubectl v1.18+](https://kubernetes.io/docs/reference/kubectl/)
 
@@ -483,6 +483,70 @@ intended to be a recommendation for all DRA drivers. Other drivers will likely
 be simpler by implementing their logic more directly than through an
 abstraction like the example driver's profiles.
 
+### Available profiles
+
+The default profile is `gpu`, which is what the quickstart above installs; all
+existing `demo/gpu-test*.yaml` fixtures continue to work unchanged. An
+additional profile exists for devices in vfio mode. This is for virtualized workloads like KubeVirt and Kata.
+
+| Profile    | Driver name             | Devices advertise                                                       | Discovery                                          | Demo fixtures              |
+|------------|-------------------------|-------------------------------------------------------------------------|----------------------------------------------------|----------------------------|
+| `gpu`      | `gpu.example.com`       | model/index/uuid                                                        | Mock (count via `--num-devices`)                   | `demo/gpu-test{1..5}.yaml` |
+| `vfio-gpu` | `vfio-gpu.example.com`  | `resource.kubernetes.io/pciBusID`, vendor/device/class, IOMMU group| Real, scans `/sys/bus/pci/drivers/vfio-pci` (vendor/device/class read from `/sys/bus/pci/devices/<BDF>`) | `demo/clusters/kind/vfio-gpu-test.yaml`   |
+
+The `vfio-gpu` profile relies on the upstream kubeletplugin framework's
+[KEP-5304][kep-5304] support to write a device metadata file at
+`/var/run/kubernetes.io/dra-device-attributes/<claim>/<request>/metadata.json`
+inside any consuming pod (enabled via the `kubeletPlugin.enableDeviceMetadata`
+Helm value / `--enable-device-metadata` CLI flag).
+
+The profile additionally injects, via the per-claim CDI spec built at
+`NodePrepareResources` time, the VFIO character devices the launcher
+needs to actually open the device: `/dev/vfio/<iommu_group>` for the
+allocated BDF and the userspace `/dev/vfio/vfio` entry point.
+
+The profile discovers devices by walking `/sys/bus/pci/drivers/vfio-pci/`,
+so every advertised device is by construction already bound to `vfio-pci`.
+No vendor/device filter or CEL selector is needed: the kernel has already
+partitioned the bus for us.
+
+Binding devices to `vfio-pci` is the operator's job (kernel cmdline
+`vfio-pci.ids=`, `driverctl set-override <BDF> vfio-pci`, a custom systemd
+unit, ...). Hosts that haven't bound anything yet will advertise an empty
+pool rather than fail the driver startup.
+
+### Installing a non-default profile
+
+```bash
+# vfio-gpu profile (real PCI passthrough for KubeVirt VMIs)
+helm upgrade -i \
+  --create-namespace \
+  --namespace dra-example-driver-vfio \
+  --set deviceProfile=vfio-gpu \
+  --set kubeletPlugin.enableDeviceMetadata=true \
+  --set driverName=vfio-gpu.example.com \
+  dra-example-driver-vfio \
+  deployments/helm/dra-example-driver
+```
+
+Each profile is a separate driver in the cluster, so both can be
+installed side-by-side without conflict.
+
+### vfio-gpu kind demo
+
+The default [`demo/clusters/kind/create-cluster.sh`](demo/clusters/kind/create-cluster.sh)
+quickstart targets the mock `gpu` profile. For vfio-gpu, prepare a Linux host with
+devices bound to `vfio-pci`, then create the cluster with vfio mounts enabled:
+
+```bash
+VFIO_GPU=true ./demo/clusters/kind/create-cluster.sh
+```
+
+Install the driver with `deviceProfile=vfio-gpu` as above, then apply
+[`demo/clusters/kind/vfio-gpu-test.yaml`](demo/clusters/kind/vfio-gpu-test.yaml)
+(KubeVirt must be installed separately). See
+[`demo/clusters/kind/README.md`](demo/clusters/kind/README.md) for the full walkthrough.
+
 ## Anatomy of a DRA resource driver
 
 TBD

api/example.com/resource/gpu/v1alpha1/api.go

@@ -31,8 +31,15 @@ const GpuConfigKind = "GpuConfig"
 type GpuConfig struct {
 	metav1.TypeMeta `json:",inline"`
 	Sharing         *GpuSharing `json:"sharing,omitempty"`
+	Mode            VfioMode    `json:"mode,omitempty"`
 }
 
+type VfioMode string
+
+const (
+	VfioModePassthrough VfioMode = "Passthrough"
+)
+
 // DefaultGpuConfig provides the default GPU configuration.
 func DefaultGpuConfig() *GpuConfig {
 	return &GpuConfig{
@@ -49,11 +56,24 @@ func DefaultGpuConfig() *GpuConfig {
 	}
 }
 
+func DefaultVfioGpuConfig() *GpuConfig {
+	return &GpuConfig{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: GroupName + "/" + Version,
+			Kind:       GpuConfigKind,
+		},
+		Mode: VfioModePassthrough,
+	}
+}
+
 // Normalize updates a GpuConfig config with implied default values based on other settings.
 func (c *GpuConfig) Normalize() error {
 	if c == nil {
 		return fmt.Errorf("config is 'nil'")
 	}
+	if c.Mode == VfioModePassthrough {
+		return nil
+	}
 	if c.Sharing == nil {
 		c.Sharing = &GpuSharing{
 			Strategy: TimeSlicingStrategy,

api/example.com/resource/gpu/v1alpha1/validate.go

@@ -67,8 +67,21 @@ func (s *GpuSharing) Validate() error {
 
 // Validate ensures that GpuConfig has a valid set of values.
 func (c *GpuConfig) Validate() error {
+	if c == nil {
+		return fmt.Errorf("GPU config is nil")
+	}
+	if c.Mode != "" {
+		return c.ValidateVfioMode()
+	}
 	if c.Sharing == nil {
 		return fmt.Errorf("no sharing strategy set")
 	}
 	return c.Sharing.Validate()
 }
+
+func (c *GpuConfig) ValidateVfioMode() error {
+	if c.Mode == VfioModePassthrough {
+		return nil
+	}
+	return fmt.Errorf("unknown GPU mode: %v", c.Mode)
+}

api/example.com/resource/gpu/v1alpha1/validate_test.go

@@ -92,6 +92,14 @@ func TestGpuConfigValidate(t *testing.T) {
 			gpuConfig: DefaultGpuConfig(),
 			expected:  nil,
 		},
+		"passthrough GpuConfig": {
+			gpuConfig: DefaultVfioGpuConfig(),
+			expected:  nil,
+		},
+		"invalid GPU mode": {
+			gpuConfig: &GpuConfig{Mode: "invalid"},
+			expected:  errors.New("unknown GPU mode: invalid"),
+		},
 		"invalid TimeSlicingConfig ignored with strategy is SpacePartitioning": {
 			gpuConfig: &GpuConfig{
 				Sharing: &GpuSharing{

cmd/dra-example-kubeletplugin/driver.go

@@ -25,6 +25,7 @@ import (
 	"k8s.io/apimachinery/pkg/types"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	coreclientset "k8s.io/client-go/kubernetes"
+	"k8s.io/dynamic-resource-allocation/api/metadata/v1alpha1"
 	"k8s.io/dynamic-resource-allocation/kubeletplugin"
 	"k8s.io/klog/v2"
 )
@@ -56,6 +57,8 @@ func NewDriver(ctx context.Context, config *Config) (*driver, error) {
 		kubeletplugin.RegistrarDirectoryPath(config.flags.kubeletRegistrarDirectoryPath),
 		kubeletplugin.PluginDataDirectoryPath(config.DriverPluginPath()),
 		kubeletplugin.RollingUpdate(types.UID(config.flags.podUID)),
+		kubeletplugin.MetadataVersions(v1alpha1.SchemeGroupVersion),
+		kubeletplugin.EnableDeviceMetadata(config.flags.enableDeviceMetadata),
 	)
 	if err != nil {
 		return nil, err
@@ -106,13 +109,22 @@ func (d *driver) prepareResourceClaim(ctx context.Context, claim *resourceapi.Re
 	}
 	var prepared []kubeletplugin.Device
 	for _, preparedDevice := range preparedDevices {
-		prepared = append(prepared, kubeletplugin.Device{
+		dev := kubeletplugin.Device{
 			Requests:     preparedDevice.GetRequestNames(),
 			PoolName:     preparedDevice.GetPoolName(),
 			DeviceName:   preparedDevice.GetDeviceName(),
 			CDIDeviceIDs: preparedDevice.GetCdiDeviceIds(),
 			ShareID:      preparedDevice.ShareID,
-		})
+		}
+
+		if allocDev, ok := d.state.allocatable[preparedDevice.GetDeviceName()]; ok && len(allocDev.Attributes) > 0 {
+			attrs := make(map[string]resourceapi.DeviceAttribute, len(allocDev.Attributes))
+			for k, v := range allocDev.Attributes {
+				attrs[string(k)] = v
+			}
+			dev.Metadata = &kubeletplugin.DeviceMetadata{Attributes: attrs}
+		}
+		prepared = append(prepared, dev)
 	}
 
 	logger.Info("Returning newly prepared devices for claim", "uid", claim.UID, "devices", prepared)

cmd/dra-example-kubeletplugin/main.go

@@ -34,6 +34,7 @@ import (
 	"sigs.k8s.io/dra-example-driver/internal/profiles"
 	"sigs.k8s.io/dra-example-driver/internal/profiles/cpu"
 	"sigs.k8s.io/dra-example-driver/internal/profiles/gpu"
+	vfiogpu "sigs.k8s.io/dra-example-driver/internal/profiles/vfio-gpu"
 	"sigs.k8s.io/dra-example-driver/pkg/flags"
 )
 
@@ -59,6 +60,7 @@ type Flags struct {
 	bindingConditions             bool
 	cpuNUMANodes                  int
 	cpusPerNUMANode               int
+	enableDeviceMetadata          bool
 }
 
 type Config struct {
@@ -76,6 +78,9 @@ var validProfiles = map[string]func(flags Flags) profiles.Profile{
 	cpu.ProfileName: func(flags Flags) profiles.Profile {
 		return cpu.NewProfile(flags.nodeName, flags.driverName, flags.cpuNUMANodes, flags.cpusPerNUMANode)
 	},
+	vfiogpu.ProfileName: func(flags Flags) profiles.Profile {
+		return vfiogpu.NewProfile(flags.nodeName, flags.driverName)
+	},
 }
 
 var validProfileNames = func() []string {
@@ -197,6 +202,13 @@ func newApp() *cli.App {
 			Destination: &flags.cpusPerNUMANode,
 			EnvVars:     []string{"CPUS_PER_NUMA_NODE"},
 		},
+		&cli.BoolFlag{
+			Name:        "enable-device-metadata",
+			Usage:       "Enable DRA in-container device metadata files for prepared devices.",
+			Value:       false,
+			Destination: &flags.enableDeviceMetadata,
+			EnvVars:     []string{"ENABLE_DEVICE_METADATA"},
+		},
 	}
 	cliFlags = append(cliFlags, flags.kubeClientConfig.Flags()...)
 	cliFlags = append(cliFlags, flags.loggingConfig.Flags()...)

cmd/dra-example-webhook/main.go

@@ -37,6 +37,7 @@ import (
 	"sigs.k8s.io/dra-example-driver/internal/profiles"
 	"sigs.k8s.io/dra-example-driver/internal/profiles/cpu"
 	"sigs.k8s.io/dra-example-driver/internal/profiles/gpu"
+	vfiogpu "sigs.k8s.io/dra-example-driver/internal/profiles/vfio-gpu"
 	"sigs.k8s.io/dra-example-driver/pkg/flags"
 )
 
@@ -53,8 +54,9 @@ type Flags struct {
 type validator func(runtime.Object) error
 
 var validProfiles = map[string]profiles.ConfigHandler{
-	gpu.ProfileName: gpu.Profile{},
-	cpu.ProfileName: cpu.Profile{},
+	gpu.ProfileName:     gpu.Profile{},
+	cpu.ProfileName:     cpu.Profile{},
+	vfiogpu.ProfileName: vfiogpu.Profile{},
 }
 
 func main() {

demo/build-driver.sh

@@ -32,9 +32,11 @@ check_demo_config || exit 1
 ${SCRIPTS_DIR}/build-driver-image.sh
 
 # If a cluster is already running, load the image onto its nodes
-EXISTING_CLUSTER="$(${KIND} get clusters | grep -w "${KIND_CLUSTER_NAME}" || true)"
-if [ "${EXISTING_CLUSTER}" != "" ]; then
-	${SCRIPTS_DIR}/load-driver-image-into-kind.sh
+if command -v kind >/dev/null; then
+	EXISTING_CLUSTER="$(${KIND} get clusters | grep -w "${KIND_CLUSTER_NAME}" || true)"
+	if [ "${EXISTING_CLUSTER}" != "" ]; then
+		${SCRIPTS_DIR}/load-driver-image-into-kind.sh
+	fi
 fi
 
 set +x

demo/clusters/README.md

@@ -11,3 +11,10 @@ common layout:
 - `delete-cluster.sh` — delete that cluster
 
 Platforms may add other scripts or notes next to these entrypoints as needed.
+
+## Available platforms
+
+| Path | Purpose |
+|---|---|
+| [`kind/`](kind/) | kind cluster for the default `gpu` (mock devices) DRA profile and, with `VFIO_GPU=true`, the `vfio-gpu` profile (host vfio-pci bindings + PCI sysfs mounts). See [`kind/README.md`](kind/README.md). |
+| [`gke/`](gke/) | GKE cluster scripts |

demo/clusters/kind/README.md

@@ -0,0 +1,101 @@
+# kind cluster
+
+Scripts to create and delete a kind cluster for the DRA demo.
+
+| File | Purpose |
+| --- | --- |
+| `create-cluster.sh` | Create the cluster (default `gpu` profile demo) |
+| `delete-cluster.sh` | Delete the cluster |
+| `kind-cluster-config-vfio.yaml` | kind config used when `VFIO_GPU=true` (PCI sysfs + `/dev/vfio` mounts) |
+| `vfio-gpu-test.yaml` | ResourceClaimTemplate for the `vfio-gpu` profile |
+
+Shared vfio helpers live in [`demo/scripts/vfio-kind.sh`](../../scripts/vfio-kind.sh).
+
+## Default (mock GPU profile)
+
+```bash
+./demo/build-driver.sh
+./demo/clusters/kind/create-cluster.sh
+helm upgrade -i \
+  --create-namespace \
+  --namespace dra-example-driver \
+  dra-example-driver \
+  deployments/helm/dra-example-driver
+```
+
+Uses the CDI-enabled kind node image built by `demo/scripts/build-kind-image.sh` and
+`demo/scripts/kind-cluster-config.yaml`.
+
+## vfio-gpu profile (`VFIO_GPU=true`)
+
+**Off by default.** Set `VFIO_GPU=true` when creating the cluster to bind-mount host
+PCI sysfs and `/dev/vfio` into kind nodes. This is required for the `vfio-gpu` driver
+profile but does **not** install the driver — you still set `deviceProfile=vfio-gpu` in
+Helm separately.
+
+**Linux host only.** The host must already have devices bound to `vfio-pci` before
+cluster creation. The script verifies bindings and exits if none are found.
+
+### Host setup
+
+Synthetic devices for testing come from [kubevirt's kind-1.35-vfio-gpu provider](https://github.com/kubevirt/kubevirt/tree/main/kubevirtci/cluster-up/cluster/kind-1.35-vfio-gpu):
+
+It has not merged yet - https://github.com/kubevirt/kubevirtci/pull/1726
+
+```bash
+sudo bash setup-host-vfio-pci.sh
+ls /sys/bus/pci/drivers/vfio-pci/   # expect BDF entries
+```
+
+Real hardware bound to vfio-pci works equally well.
+
+### Cluster + driver
+
+```bash
+# 1. Cluster (vfio mounts into nodes)
+VFIO_GPU=true ./demo/clusters/kind/create-cluster.sh
+
+# 2. Driver image (build from this repo; published images may predate vfio-gpu)
+./demo/build-driver.sh
+
+# 3. Driver install (vfio-gpu profile)
+helm upgrade --install \
+    --create-namespace \
+    --namespace dra-example-driver-vfio \
+    --set deviceProfile=vfio-gpu \
+    --set kubeletPlugin.enableDeviceMetadata=true \
+    --set driverName=vfio-gpu.example.com \
+    dra-example-driver-vfio \
+    deployments/helm/dra-example-driver
+
+Verify the driver:
+
+```bash
+kubectl -n dra-example-driver-vfio get ds -o jsonpath='{range .spec.template.spec.containers[?(@.name=="plugin")].env[?(@.name=="DEVICE_PROFILE")]}{.value}{"\n"}{end}'
+kubectl get resourceslices -o custom-columns='NAME:.metadata.name,DRIVER:.spec.driver,NODE:.spec.nodeName'
+```
+
+Tear down:
+
+```bash
+./demo/clusters/kind/delete-cluster.sh
+```
+
+### Environment variables
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `VFIO_GPU` | `false` | Enable vfio-gpu cluster mode (`true` / `1` / `yes` / `on`) |
+| `VFIO_KIND_NODE_IMAGE` | pinned `kindest/node:v1.35.0` | kind node image when `VFIO_GPU=true` |
+| `VFIO_KIND_CLUSTER_CONFIG_PATH` | `kind-cluster-config-vfio.yaml` in this directory | kind config when `VFIO_GPU=true` |
+
+Other variables (`KIND_CLUSTER_NAME`, `CONTAINER_TOOL`, …) come from [`demo/scripts/common.sh`](../../scripts/common.sh).
+
+### Two knobs (cluster vs driver)
+
+| Setting | Layer | What it does |
+| --- | --- | --- |
+| `VFIO_GPU=true` | Cluster (`create-cluster.sh`) | vfio-pci preflight, vfio kind config, post-create node setup |
+| `deviceProfile=vfio-gpu` | Driver (Helm) | Driver discovers vfio-bound PCI devices and prepares `/dev/vfio` CDI |
+
+Both are required for the end-to-end vfio-gpu demo.