Skip to content

Commit acb4b99

Browse files
beraldolealbutler54
beraldoleal
and
butler54
authored
coco: initial integration for Confidential Containers and Trustee operators (#80)
* coco: initial integration with ztvp This adds initial integration for Confidential Containers and Trustee Operators as a separated clustergroup. Co-authored-by: Chris Butler <chris.butler@redhat.com> Signed-off-by: Beraldo Leal <bleal@redhat.com> * coco: add imperative job to configure x509pop Add automated configuration for SPIRE Server x509pop NodeAttestor plugin required for CoCo peer-pods attestation. CoCo peer-pods run on untrusted cloud infrastructure. Using k8s_psat would require trusting the cloud provider's cluster. Instead, pods perform hardware TEE attestation to KBS to obtain x509 certificates as cryptographic proof of running in genuine confidential hardware, then use x509pop to register with SPIRE. The Red Hat SPIRE Operator's SpireServer CRD does not expose x509pop configuration, requiring a ConfigMap patch via this imperative job. Signed-off-by: Beraldo Leal <bleal@redhat.com> * coco: introducing the hello-coco app Add hello-coco Helm chart demonstrating SPIRE agent deployment in confidential containers using x509pop node attestation. The chart deploys a test pod in a CoCo peer-pod (confidential VM with AMD SNP or Intel TDX) that fetches SPIRE agent certificates from KBS after TEE attestation, establishing hardware as the root of trust instead of Kubernetes. The pod contains three containers: init container fetches sealed secrets from KBS, SPIRE agent uses x509pop for node attestation, and test workload receives SPIFFE SVIDs via unix attestation. This validates the complete integration flow between ZTVP and CoCo components. Note: This could be dropped, if we stick with only the todoapp. Signed-off-by: Beraldo Leal <bleal@redhat.com> * coco: update the values-secret template Signed-off-by: Beraldo Leal <bleal@redhat.com> * coco: add get-pcr.sh script for attestation measurements * coco: add get-secrets-coco.sh Signed-off-by: Beraldo Leal <bleal@redhat.com> * coco: adding confidential documentation Basic markdown file with deployment steps. Signed-off-by: Beraldo Leal <bleal@redhat.com> * coco: automate pull-secret via ESO cross-namespace Peer-pods don't have access to the node's pull-secret, needed for private repos. Use ESO kubernetes provider to sync pull-secret from openshift-config to the workload namespace. Signed-off-by: Beraldo Leal <bleal@redhat.com> --------- Signed-off-by: Beraldo Leal <bleal@redhat.com> Co-authored-by: Chris Butler <chris.butler@redhat.com>
1 parent 3ef9ab6 commit acb4b99

23 files changed

Lines changed: 2134 additions & 0 deletions

‎ansible/azure-nat-gateway.yaml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
---
2+
3+
- name: Configure Azure NAT Gateway
4+
become: false
5+
connection: local
6+
hosts: localhost
7+
gather_facts: false
8+
vars:
9+
kubeconfig: "{{ lookup('env', 'KUBECONFIG') }}"
10+
resource_prefix: "coco"
11+
tasks:
12+
- name: Get Azure credentials # noqa: syntax-check[unknown-module]
13+
kubernetes.core.k8s_info:
14+
kind: Secret
15+
namespace: openshift-cloud-controller-manager
16+
name: azure-cloud-credentials
17+
register: azure_credentials
18+
retries: 20
19+
delay: 5
20+
21+
- name: Get Azure configuration # noqa: syntax-check[unknown-module]
22+
kubernetes.core.k8s_info:
23+
kind: ConfigMap
24+
namespace: openshift-cloud-controller-manager
25+
name: cloud-conf
26+
register: azure_cloud_conf
27+
retries: 20
28+
delay: 5
29+
30+
- name: Set facts
31+
ansible.builtin.set_fact:
32+
azure_subscription_id: "{{ (azure_cloud_conf.resources[0]['data']['cloud.conf'] | from_json)['subscriptionId'] }}"
33+
azure_tenant_id: "{{ (azure_cloud_conf.resources[0]['data']['cloud.conf'] | from_json)['tenantId'] }}"
34+
azure_resource_group: "{{ (azure_cloud_conf.resources[0]['data']['cloud.conf'] | from_json)['vnetResourceGroup'] }}"
35+
azure_client_id: "{{ azure_credentials.resources[0]['data']['azure_client_id'] | b64decode }}"
36+
azure_client_secret: "{{ azure_credentials.resources[0]['data']['azure_client_secret'] | b64decode }}"
37+
azure_vnet: "{{ (azure_cloud_conf.resources[0]['data']['cloud.conf'] | from_json)['vnetName'] }}"
38+
azure_subnet: "{{ (azure_cloud_conf.resources[0]['data']['cloud.conf'] | from_json)['subnetName'] }}"
39+
coco_public_ip_name: "{{ resource_prefix }}-pip"
40+
coco_nat_gateway_name: "{{ resource_prefix }}-nat-gateway"
41+
no_log: true
42+
43+
- name: Create Public IP for NAT Gateway
44+
azure.azcollection.azure_rm_publicipaddress:
45+
subscription_id: "{{ azure_subscription_id }}"
46+
tenant: "{{ azure_tenant_id }}"
47+
client_id: "{{ azure_client_id }}"
48+
secret: "{{ azure_client_secret }}"
49+
resource_group: "{{ azure_resource_group }}"
50+
name: "{{ coco_public_ip_name }}"
51+
sku: "standard"
52+
allocation_method: "static"
53+
54+
- name: Retrieve Public IP for NAT Gateway
55+
azure.azcollection.azure_rm_publicipaddress_info:
56+
subscription_id: "{{ azure_subscription_id }}"
57+
tenant: "{{ azure_tenant_id }}"
58+
client_id: "{{ azure_client_id }}"
59+
secret: "{{ azure_client_secret }}"
60+
resource_group: "{{ azure_resource_group }}"
61+
name: "{{ coco_public_ip_name }}"
62+
register: coco_gw_public_ip
63+
64+
- name: Create NAT Gateway
65+
azure.azcollection.azure_rm_natgateway:
66+
subscription_id: "{{ azure_subscription_id }}"
67+
tenant: "{{ azure_tenant_id }}"
68+
client_id: "{{ azure_client_id }}"
69+
secret: "{{ azure_client_secret }}"
70+
resource_group: "{{ azure_resource_group }}"
71+
name: "{{ coco_nat_gateway_name }}"
72+
idle_timeout_in_minutes: 10
73+
sku:
74+
name: standard
75+
public_ip_addresses:
76+
- "{{ coco_gw_public_ip.publicipaddresses[0].id }}"
77+
register: coco_natgw
78+
79+
- name: Update the worker subnet to associate NAT gateway
80+
azure.azcollection.azure_rm_subnet:
81+
subscription_id: "{{ azure_subscription_id }}"
82+
tenant: "{{ azure_tenant_id }}"
83+
client_id: "{{ azure_client_id }}"
84+
secret: "{{ azure_client_secret }}"
85+
resource_group: "{{ azure_resource_group }}"
86+
name: "{{ azure_subnet }}"
87+
virtual_network_name: "{{ azure_vnet }}"
88+
nat_gateway: "{{ coco_nat_gateway_name }}"

‎ansible/configure-spire-server-x509pop.yaml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
# Configure SPIRE Server to support x509pop node attestation for CoCo pods
3+
# The Red Hat SPIRE Operator's SpireServer CRD does not expose x509pop plugin configuration
4+
# This job patches the operator-generated ConfigMap and StatefulSet to add x509pop support
5+
#
6+
# IMPORTANT: The operator must have CREATE_ONLY_MODE=true env var set (via subscription
7+
# config) to prevent it from reverting our manual patches. Without this, the operator
8+
# continuously reconciles and overwrites x509pop changes.
9+
# Note: In v0.2.0 (tech-preview) this was done via a CR annotation
10+
# (ztwim.openshift.io/create-only). In v1.0.0 (GA) it changed to the env var.
11+
12+
- name: Configure SPIRE Server for x509pop attestation
13+
become: false
14+
connection: local
15+
hosts: localhost
16+
gather_facts: false
17+
vars:
18+
spire_namespace: "zero-trust-workload-identity-manager"
19+
configmap_name: "spire-server"
20+
statefulset_name: "spire-server"
21+
ca_configmap_name: "spire-x509pop-ca"
22+
ca_mount_path: "/run/spire/x509pop-ca"
23+
tasks:
24+
- name: Get ZeroTrustWorkloadIdentityManager CR to determine expected cluster name # noqa: syntax-check[unknown-module]
25+
kubernetes.core.k8s_info:
26+
api_version: operator.openshift.io/v1alpha1
27+
kind: ZeroTrustWorkloadIdentityManager
28+
name: cluster
29+
register: ztwim_cr
30+
retries: 30
31+
delay: 10
32+
until: ztwim_cr.resources | length > 0
33+
34+
- name: Extract expected cluster name from ZTWIM CR
35+
ansible.builtin.set_fact:
36+
expected_cluster_name: "{{ ztwim_cr.resources[0].spec.clusterName }}"
37+
38+
- name: Display expected cluster name
39+
ansible.builtin.debug:
40+
msg: "Expected cluster name from ZTWIM CR: {{ expected_cluster_name }}"
41+
42+
- name: Wait for SPIRE Server ConfigMap with correct cluster name # noqa: syntax-check[unknown-module]
43+
kubernetes.core.k8s_info:
44+
kind: ConfigMap
45+
namespace: "{{ spire_namespace }}"
46+
name: "{{ configmap_name }}"
47+
register: spire_configmap
48+
retries: 60
49+
delay: 5
50+
until: >
51+
spire_configmap.resources | length > 0 and
52+
(spire_configmap.resources[0].data['server.conf'] | from_json
53+
).plugins.NodeAttestor[0].k8s_psat.plugin_data.clusters[0][expected_cluster_name] is defined
54+
55+
- name: ConfigMap has correct cluster name
56+
ansible.builtin.debug:
57+
msg: "ConfigMap verified with correct cluster name: {{ expected_cluster_name }}"
58+
59+
- name: Get current SPIRE Server configuration # noqa: syntax-check[unknown-module]
60+
kubernetes.core.k8s_info:
61+
kind: ConfigMap
62+
namespace: "{{ spire_namespace }}"
63+
name: "{{ configmap_name }}"
64+
register: spire_config
65+
66+
- name: Parse server configuration
67+
ansible.builtin.set_fact:
68+
server_conf: "{{ spire_config.resources[0].data['server.conf'] | from_json }}"
69+
70+
- name: Check if x509pop already configured
71+
ansible.builtin.set_fact:
72+
x509pop_exists: "{{ server_conf.plugins.NodeAttestor | selectattr('x509pop', 'defined') | list | length > 0 }}"
73+
74+
- name: Add x509pop NodeAttestor plugin # noqa: syntax-check[unknown-module]
75+
kubernetes.core.k8s:
76+
state: present
77+
definition:
78+
apiVersion: v1
79+
kind: ConfigMap
80+
metadata:
81+
name: "{{ configmap_name }}"
82+
namespace: "{{ spire_namespace }}"
83+
data:
84+
server.conf: >-
85+
{{ server_conf | combine({'plugins': {'NodeAttestor':
86+
server_conf.plugins.NodeAttestor + [{'x509pop': {'plugin_data':
87+
{'ca_bundle_path': '/run/spire/x509pop-ca/ca-bundle.pem'}}}]}},
88+
recursive=True) | to_json }}
89+
when: not x509pop_exists
90+
91+
- name: Wait for SPIRE Server StatefulSet to exist # noqa: syntax-check[unknown-module]
92+
kubernetes.core.k8s_info:
93+
kind: StatefulSet
94+
namespace: "{{ spire_namespace }}"
95+
name: "{{ statefulset_name }}"
96+
register: spire_statefulset
97+
retries: 30
98+
delay: 10
99+
until: spire_statefulset.resources | length > 0
100+
101+
- name: Check if CA volume already mounted
102+
ansible.builtin.set_fact:
103+
ca_volume_exists: "{{ spire_statefulset.resources[0].spec.template.spec.volumes | selectattr('name', 'equalto', 'x509pop-ca') | list | length > 0 }}"
104+
105+
- name: Add CA volume to SPIRE Server StatefulSet # noqa: syntax-check[unknown-module]
106+
kubernetes.core.k8s:
107+
state: patched
108+
kind: StatefulSet
109+
namespace: "{{ spire_namespace }}"
110+
name: "{{ statefulset_name }}"
111+
definition:
112+
spec:
113+
template:
114+
spec:
115+
volumes:
116+
- name: x509pop-ca
117+
configMap:
118+
name: "{{ ca_configmap_name }}"
119+
containers:
120+
- name: spire-server
121+
volumeMounts:
122+
- name: x509pop-ca
123+
mountPath: "{{ ca_mount_path }}"
124+
readOnly: true
125+
when: not ca_volume_exists
126+
127+
- name: Restart SPIRE Server to apply configuration # noqa: syntax-check[unknown-module]
128+
kubernetes.core.k8s:
129+
state: absent
130+
kind: Pod
131+
namespace: "{{ spire_namespace }}"
132+
label_selectors:
133+
- app.kubernetes.io/name=server
134+
when: (not x509pop_exists) or (not ca_volume_exists)
135+
136+
- name: Configuration status
137+
ansible.builtin.debug:
138+
msg: >-
139+
{{ 'x509pop already configured' if (x509pop_exists and ca_volume_exists)
140+
else 'x509pop NodeAttestor plugin and CA volume mount configured successfully' }}
141+
142+
- name: Final status
143+
ansible.builtin.debug:
144+
msg: "x509pop configuration complete. CREATE_ONLY_MODE env var on the operator prevents reverts."

0 commit comments

Comments
 (0)