feat: add bare metal support for Intel TDX and AMD SEV-SNP

README.md

@@ -2,21 +2,23 @@
 
 Validated pattern for deploying confidential containers on OpenShift using the [Validated Patterns](https://validatedpatterns.io/) framework.
 
-Confidential containers use hardware-backed Trusted Execution Environments (TEEs) to isolate workloads from cluster and hypervisor administrators. This pattern deploys and configures the Red Hat CoCo stack — including the sandboxed containers operator, Trustee (Key Broker Service), and peer-pod infrastructure — on Azure.
+Confidential containers use hardware-backed Trusted Execution Environments (TEEs) to isolate workloads from cluster and hypervisor administrators. This pattern deploys and configures the Red Hat CoCo stack — including the sandboxed containers operator, Trustee (Key Broker Service), and peer-pod infrastructure — on Azure and bare metal.
 
 ## Topologies
 
-The pattern provides two deployment topologies:
+The pattern provides three deployment topologies:
 
-1. **Single cluster** (`simple` clusterGroup) — deploys all components (Trustee, Vault, ACM, sandboxed containers, workloads) in one cluster. This breaks the RACI separation expected in a remote attestation architecture but simplifies testing and demonstrations.
+1. **Single cluster** (`simple` clusterGroup) — deploys all components (Trustee, Vault, ACM, sandboxed containers, workloads) in one cluster on Azure. This breaks the RACI separation expected in a remote attestation architecture but simplifies testing and demonstrations.
 
 2. **Multi-cluster** (`trusted-hub` + `spoke` clusterGroups) — separates the trusted zone from the untrusted workload zone:
    - **Hub** (`trusted-hub`): Runs Trustee (KBS + attestation service), HashiCorp Vault, ACM, and cert-manager. This cluster is the trust anchor.
    - **Spoke** (`spoke`): Runs the sandboxed containers operator and confidential workloads. The spoke is imported into ACM and managed from the hub.
 
+3. **Bare metal** (`baremetal` clusterGroup) — deploys all components on bare metal hardware with Intel TDX or AMD SEV-SNP support. NFD (Node Feature Discovery) auto-detects the CPU architecture and configures the appropriate runtime. Supports SNO (Single Node OpenShift) and multi-node clusters.
+
 The topology is controlled by the `main.clusterGroupName` field in `values-global.yaml`.
 
-Currently supports Azure via peer-pods. Peer-pods provision confidential VMs (`Standard_DCas_v5` family) directly on the Azure hypervisor rather than nesting VMs inside worker nodes.
+Azure deployments use peer-pods, which provision confidential VMs (`Standard_DCas_v5` family) directly on the Azure hypervisor. Bare metal deployments use layered images and hardware TEE features directly.
 
 ## Current version (4.*)
 
@@ -42,9 +44,21 @@ All previous versions used pre-GA (Technology Preview) releases of Trustee:
 
 ### Prerequisites
 
+**Azure deployments:**
+
 - OpenShift 4.17+ cluster on Azure (self-managed via `openshift-install` or ARO)
 - Azure `Standard_DCas_v5` VM quota in your target region (these are confidential computing VMs and are not available in all regions). See the note below for more details.
 - Azure DNS hosting the cluster's DNS zone
+
+**Bare metal deployments:**
+
+- OpenShift 4.17+ cluster on bare metal with Intel TDX or AMD SEV-SNP hardware
+- BIOS/firmware configured to enable TDX or SEV-SNP
+- Available block devices for LVMS storage (auto-discovered)
+- For Intel TDX: an Intel PCS API key from [api.portal.trustedservices.intel.com](https://api.portal.trustedservices.intel.com/)
+
+**Common:**
+
 - Tools on your workstation: `podman`, `yq`, `jq`, `skopeo`
 - OpenShift pull secret saved at `~/pull-secret.json` (download from [console.redhat.com](https://console.redhat.com/openshift/downloads))
 - Fork the repository — ArgoCD reconciles cluster state against your fork, so changes must be pushed to your remote
@@ -53,20 +67,20 @@ All previous versions used pre-GA (Technology Preview) releases of Trustee:
 
 These scripts generate the cryptographic material and attestation measurements needed by Trustee and the peer-pod VMs. Run them once before your first deployment.
 
-1. `bash scripts/gen-secrets.sh` — generates KBS key pairs, attestation policy seeds, and copies `values-secret.yaml.template` to `~/values-secret-coco-pattern.yaml`
-2. `bash scripts/get-pcr.sh` — retrieves PCR measurements from the peer-pod VM image and stores them at `~/.coco-pattern/measurements.json` (requires `podman`, `skopeo`, and `~/pull-secret.json`)
-3. Review and customise `~/values-secret-coco-pattern.yaml` — this file is loaded into Vault and provides secrets to the pattern
+1. `bash scripts/gen-secrets.sh` — generates KBS key pairs, PCCS certificates/tokens (for bare metal), and copies `values-secret.yaml.template` to `~/values-secret-coco-pattern.yaml`
+2. `bash scripts/get-pcr.sh` — retrieves PCR measurements from the peer-pod VM image and stores them at `~/.coco-pattern/measurements.json` (requires `podman`, `skopeo`, and `~/pull-secret.json`). **Not required for bare metal deployments.**
+3. Review and customise `~/values-secret-coco-pattern.yaml` — this file is loaded into Vault and provides secrets to the pattern. For bare metal, uncomment the PCCS secrets section and provide your Intel PCS API key.
 
 > **Note:** `gen-secrets.sh` will not overwrite existing secrets. Delete `~/.coco-pattern/` if you need to regenerate.
 
-### Single cluster deployment
+### Single cluster deployment (Azure)
 
 1. Set `main.clusterGroupName: simple` in `values-global.yaml`
 2. Ensure your Azure configuration is populated in `values-global.yaml` (see `global.azure.*` fields)
 3. `./pattern.sh make install`
 4. Wait for the cluster to reboot all nodes (the sandboxed containers operator triggers a MachineConfig update). Monitor progress in the ArgoCD UI.
 
-### Multi-cluster deployment
+### Multi-cluster deployment (Azure)
 
 1. Set `main.clusterGroupName: trusted-hub` in `values-global.yaml`
 2. Deploy the hub cluster: `./pattern.sh make install`
@@ -76,6 +90,25 @@ These scripts generate the cryptographic material and attestation measurements n
    (see [importing a cluster](https://validatedpatterns.io/learn/importing-a-cluster/))
 6. ACM will automatically deploy the `spoke` clusterGroup applications (sandboxed containers, workloads) to the imported cluster
 
+### Bare metal deployment
+
+1. Set `main.clusterGroupName: baremetal` in `values-global.yaml`
+2. Run `bash scripts/gen-secrets.sh` to generate KBS keys and PCCS secrets
+3. For Intel TDX: uncomment the PCCS secrets in `~/values-secret-coco-pattern.yaml` and provide your Intel PCS API key
+4. `./pattern.sh make install`
+5. Wait for the cluster to reboot nodes (MachineConfig updates for TDX kernel parameters and vsock)
+
+The system auto-detects your hardware:
+
+- **NFD** discovers Intel TDX or AMD SEV-SNP capabilities and labels nodes
+- **LVMS** auto-discovers available block devices for storage
+- **RuntimeClass** `kata-cc` is created automatically pointing to the correct handler (`kata-tdx` or `kata-snp`)
+- Both `kata-tdx` and `kata-snp` RuntimeClasses are deployed; only the one matching your hardware has schedulable nodes
+- MachineConfigs are deployed for both `master` and `worker` roles (safe on SNO where only master exists)
+- PCCS and QGS services deploy unconditionally; DaemonSets only schedule on Intel nodes via NFD labels
+
+Optional: pin PCCS to a specific node with `bash scripts/get-pccs-node.sh` and set `baremetal.pccs.nodeSelector` in the baremetal chart values.
+
 ## Sample applications
 
 Two sample applications are deployed on the cluster running confidential workloads (the single cluster in `simple` mode, or the spoke in multi-cluster mode):

ansible/detect-runtime-class.yaml

@@ -0,0 +1,56 @@
+- name: Detect and configure runtime class
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  tasks:
+    - name: Check for Intel TDX nodes
+      kubernetes.core.k8s_info:
+        api_version: v1
+        kind: Node
+        label_selectors:
+          - intel.feature.node.kubernetes.io/tdx=true
+      register: tdx_nodes
+
+    - name: Check for AMD SEV-SNP nodes
+      kubernetes.core.k8s_info:
+        api_version: v1
+        kind: Node
+        label_selectors:
+          - amd.feature.node.kubernetes.io/snp=true
+      register: snp_nodes
+
+    - name: Set runtime handler for Intel TDX
+      ansible.builtin.set_fact:
+        kata_handler: "kata-tdx"
+        kata_overhead:
+          memory: "350Mi"
+          cpu: "250m"
+          tdx.intel.com/keys: "1"
+        kata_node_selector:
+          intel.feature.node.kubernetes.io/tdx: "true"
+      when: tdx_nodes.resources | length > 0
+
+    - name: Set runtime handler for AMD SEV-SNP
+      ansible.builtin.set_fact:
+        kata_handler: "kata-snp"
+        kata_overhead:
+          memory: "350Mi"
+          cpu: "250m"
+        kata_node_selector:
+          amd.feature.node.kubernetes.io/snp: "true"
+      when: snp_nodes.resources | length > 0
+
+    - name: Create kata-cc RuntimeClass
+      kubernetes.core.k8s:
+        state: present
+        definition:
+          apiVersion: node.k8s.io/v1
+          kind: RuntimeClass
+          metadata:
+            name: kata-cc
+          handler: "{{ kata_handler }}"
+          overhead:
+            podFixed: "{{ kata_overhead }}"
+          scheduling:
+            nodeSelector: "{{ kata_node_selector }}"
+      when: kata_handler is defined

charts/all/baremetal/Chart.yaml

@@ -0,0 +1,9 @@
+apiVersion: v2
+description: Bare metal platform configuration (NFD rules, MachineConfigs, RuntimeClasses, Intel device plugin).
+keywords:
+- pattern
+- upstream
+- sandbox
+- baremetal
+name: baremetal
+version: 0.0.1

charts/all/baremetal/bm-kernel-params.yaml

@@ -0,0 +1,2 @@
+[hypervisor.qemu]
+kernel_params="agent.aa_kbc_params=cc_kbc::http://kbs-trustee-operator-system.{{ .Values.global.hubClusterDomain }}"

charts/all/baremetal/templates/kata-nfd.yaml

@@ -0,0 +1,80 @@
+apiVersion: nfd.openshift.io/v1alpha1
+kind: NodeFeatureRule
+metadata:
+  name: consolidated-hardware-features
+  namespace: openshift-nfd
+spec:
+  rules:
+    - name: "runtime.kata"
+      labels:
+        feature.node.kubernetes.io/runtime.kata: "true"
+      matchAny:
+        - matchFeatures:
+            - feature: cpu.cpuid
+              matchExpressions:
+                SSE42: { op: Exists }
+                VMX: { op: Exists }
+            - feature: kernel.loadedmodule
+              matchExpressions:
+                kvm: { op: Exists }
+                kvm_intel: { op: Exists }
+        - matchFeatures:
+            - feature: cpu.cpuid
+              matchExpressions:
+                SSE42: { op: Exists }
+                SVM: { op: Exists }
+            - feature: kernel.loadedmodule
+              matchExpressions:
+                kvm: { op: Exists }
+                kvm_amd: { op: Exists }
+
+    - name: "amd.sev-snp"
+      labels:
+        amd.feature.node.kubernetes.io/snp: "true"
+      extendedResources:
+        sev-snp.amd.com/esids: "@cpu.security.sev.encrypted_state_ids"
+      matchFeatures:
+        - feature: cpu.cpuid
+          matchExpressions:
+            SVM: { op: Exists }
+        - feature: cpu.security
+          matchExpressions:
+            sev.snp.enabled: { op: Exists }
+
+    - name: "intel.sgx"
+      labels:
+        intel.feature.node.kubernetes.io/sgx: "true"
+      extendedResources:
+        sgx.intel.com/epc: "@cpu.security.sgx.epc"
+      matchFeatures:
+        - feature: cpu.cpuid
+          matchExpressions:
+            SGX: { op: Exists }
+            SGXLC: { op: Exists }
+        - feature: cpu.security
+          matchExpressions:
+            sgx.enabled: { op: IsTrue }
+        - feature: kernel.config
+          matchExpressions:
+            X86_SGX: { op: Exists }
+
+    - name: "intel.tdx"
+      labels:
+        intel.feature.node.kubernetes.io/tdx: "true"
+      extendedResources:
+        tdx.intel.com/keys: "@cpu.security.tdx.total_keys"
+      matchFeatures:
+        - feature: cpu.cpuid
+          matchExpressions:
+            VMX: { op: Exists }
+        - feature: cpu.security
+          matchExpressions:
+            tdx.enabled: { op: Exists }
+
+    - name: "ibm.se.enabled"
+      labels:
+        ibm.feature.node.kubernetes.io/se: "true"
+      matchFeatures:
+        - feature: cpu.security
+          matchExpressions:
+            se.enabled: { op: IsTrue }

charts/all/baremetal/templates/kernel-params-mco.yaml

@@ -0,0 +1,21 @@
+{{- range list "master" "worker" }}
+---
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: {{ . }}
+  name: 96-kata-kernel-config-{{ . }}
+  namespace: openshift-machine-config-operator
+spec:
+  config:
+    ignition:
+      version: 3.2.0
+    storage:
+      files:
+      - contents:
+          source: 'data:text/plain;charset=utf-8;base64,{{ tpl ($.Files.Get "bm-kernel-params.yaml") $ | b64enc }}'
+        mode: 420
+        overwrite: true
+        path: /etc/kata-containers/snp/config.d/96-kata-kernel-config
+{{- end }}

charts/all/baremetal/templates/nfd-instance.yaml

@@ -0,0 +1,12 @@
+apiVersion: nfd.openshift.io/v1
+kind: NodeFeatureDiscovery
+metadata:
+  name: nfd-instance
+  namespace: openshift-nfd
+spec:
+  operand:
+    image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.20
+    imagePullPolicy: Always
+    servicePort: 12000
+  workerConfig:
+    configData: |

charts/all/baremetal/templates/runtimeclass-amd-snp.yaml

@@ -0,0 +1,12 @@
+# apiVersion: node.k8s.io/v1
+# kind: RuntimeClass
+# metadata:
+#   name: kata-snp
+# handler: kata-snp
+# overhead:
+#   podFixed:
+#     memory: "350Mi"
+#     cpu: "250m"
+# scheduling:
+#   nodeSelector:
+#     amd.feature.node.kubernetes.io/snp: "true"

charts/all/baremetal/templates/runtimeclass-intel-tdx.yaml

@@ -0,0 +1,13 @@
+# apiVersion: node.k8s.io/v1
+# kind: RuntimeClass
+# metadata:
+#   name: kata-tdx
+# handler: kata-tdx
+# overhead:
+#   podFixed:
+#     memory: "350Mi"
+#     cpu: "250m"
+#     tdx.intel.com/keys: 1
+# scheduling:
+#   nodeSelector:
+#     intel.feature.node.kubernetes.io/tdx: "true"

charts/all/baremetal/templates/vsock-mco.yaml

@@ -0,0 +1,21 @@
+{{- range list "master" "worker" }}
+---
+apiVersion: machineconfiguration.openshift.io/v1
+kind: MachineConfig
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: {{ . }}
+  name: 99-enable-coco-{{ . }}
+spec:
+  kernelArguments:
+  - nohibernate
+  config:
+    ignition:
+      version: 3.2.0
+    storage:
+      files:
+        - path: /etc/modules-load.d/vsock.conf
+          mode: 0644
+          contents:
+            source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK
+{{- end }}

charts/all/baremetal/values.yaml

@@ -0,0 +1 @@
+# No configurable values — platform resources use fixed configurations.

charts/all/intel-dcap/Chart.yaml

@@ -0,0 +1,10 @@
+apiVersion: v2
+description: Intel DCAP services (PCCS and QGS) for TDX remote attestation.
+keywords:
+- pattern
+- intel
+- tdx
+- pccs
+- qgs
+name: intel-dcap
+version: 0.0.1

charts/all/intel-dcap/templates/intel-dpo-sgx.yaml

@@ -0,0 +1,11 @@
+apiVersion: deviceplugin.intel.com/v1
+kind: SgxDevicePlugin
+metadata:
+  name: sgxdeviceplugin-sample
+spec:
+  image: registry.connect.redhat.com/intel/intel-sgx-plugin@sha256:f2c77521c6dae6b4db1896a5784ba8b06a5ebb2a01684184fc90143cfcca7bf4
+  enclaveLimit: 110
+  provisionLimit: 110
+  logLevel: 4
+  nodeSelector:
+    intel.feature.node.kubernetes.io/sgx: "true"