Skip to content

Commit f860575

Browse files
bogdandobogdando
committed
nova05: separate pre and post ceph OSCP configs
HCI uses edpm post ceph to also patch the OSCP which looks suboptimal. Separate this into a two steps: * make post-ceph nodeset layer handling EDPM resources only (NodeSet, Services, Secrets, ConfigMap) - remove OSCP layer resources * add control-plane-post-ceph layer for OSCP with Glance RBD + Ceph extraMounts and secret config. Instead of re-including lib/control-plane (which required network-values duplication) provide only required values for lib/control-plane/base. * update README.md stages list from 4 to 6 steps, automation step count from 9 to 10 * add note for defferred Ceph config for control-plane.md * add control-plane post-ceph doc for the dedicated Ceph control-plane update stage * remove control-plane update references for dataplane-post-ceph.md, add prerequisite for control-plane-post-ceph * add control-plane-post-ceph stage between pre-ceph deployment and post-ceph nodeset ini automation vars Signed-off-by: Bohdan Dobrelia <bdobreli@redhat.com>
1 parent c1a6e5c commit f860575

15 files changed

Lines changed: 854 additions & 70 deletions

File tree

‎automation/vars/nova05epsilon.yaml‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,14 +101,25 @@ vas:
101101
type: playbook
102102
source: "../../hooks/playbooks/ceph.yml"
103103
inventory: "${HOME}/ci-framework-data/artifacts/ceph_inventory.yml"
104-
105-
- name: edpm-nodeset-post-ceph
106-
path: examples/dt/nova/nova05epsilon
104+
- name: control-plane-post-ceph
105+
path: examples/dt/nova/nova05epsilon/control-plane-post-ceph
107106
wait_conditions:
108107
- >-
109108
oc -n openstack wait osctlplane controlplane
110109
--for condition=Ready
111110
--timeout=20m
111+
values:
112+
- name: network-values
113+
src_file: network-values.yaml
114+
- name: service-values
115+
src_file: service-values.yaml
116+
- name: edpm-nodeset-values-post-ceph
117+
src_file: values.yaml
118+
build_output: control-plane-post-ceph.yaml
119+
120+
- name: edpm-nodeset-post-ceph
121+
path: examples/dt/nova/nova05epsilon
122+
wait_conditions:
112123
- >-
113124
oc -n openstack wait osdpns
114125
gpu-computes-edpm

‎dt/nova/nova05epsilon/control-plane-post-ceph/ceph_secret.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
data:
4+
ceph.client.openstack.keyring: _ignored_
5+
ceph.conf: _ignored_
6+
kind: Secret
7+
metadata:
8+
name: ceph-conf-files
9+
namespace: openstack
10+
type: Opaque

‎dt/nova/nova05epsilon/control-plane-post-ceph/kustomization.yaml‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1alpha1
3+
kind: Component
4+
5+
transformers:
6+
- |-
7+
apiVersion: builtin
8+
kind: NamespaceTransformer
9+
metadata:
10+
name: _ignored_
11+
namespace: openstack
12+
setRoleBindingSubjects: none
13+
unsetOnly: true
14+
fieldSpecs:
15+
- path: metadata/name
16+
kind: Namespace
17+
create: true
18+
19+
components:
20+
- ../../../../lib/control-plane/base
21+
- ../../../../lib/control-plane/service-endpoints
22+
- ../../../../lib/control-plane/dns
23+
- ../../../../lib/control-plane/storage
24+
- ../../../../lib/control-plane/ovn-bridge
25+
- ../../../../lib/control-plane/job-settings
26+
- ../../../../lib/control-plane/tls
27+
- ../../../../lib/control-plane/messaging-bus
28+
29+
resources:
30+
- ceph_secret.yaml
31+
32+
replacements:
33+
# Ceph config files -> Secret
34+
- source:
35+
kind: ConfigMap
36+
name: edpm-nodeset-values-post-ceph
37+
fieldPath: data.ceph_conf
38+
targets:
39+
- select:
40+
kind: Secret
41+
name: ceph-conf-files
42+
fieldPaths:
43+
- data
44+
options:
45+
create: true
46+
# Glance RBD backend
47+
- source:
48+
kind: ConfigMap
49+
name: service-values
50+
fieldPath: data.glance.customServiceConfig
51+
targets:
52+
- select:
53+
kind: OpenStackControlPlane
54+
fieldPaths:
55+
- spec.glance.template.customServiceConfig
56+
options:
57+
create: true
58+
- source:
59+
kind: ConfigMap
60+
name: service-values
61+
fieldPath: data.glance.default.replicas
62+
targets:
63+
- select:
64+
kind: OpenStackControlPlane
65+
fieldPaths:
66+
- spec.glance.template.glanceAPIs.default.replicas
67+
options:
68+
create: true
69+
- source:
70+
kind: ConfigMap
71+
name: service-values
72+
fieldPath: data.extraMounts
73+
targets:
74+
- select:
75+
kind: OpenStackControlPlane
76+
fieldPaths:
77+
- spec.extraMounts
78+
options:
79+
create: true
80+
# Swift
81+
- source:
82+
kind: ConfigMap
83+
name: service-values
84+
fieldPath: data.swift.enabled
85+
targets:
86+
- select:
87+
kind: OpenStackControlPlane
88+
fieldPaths:
89+
- spec.swift.enabled
90+
options:
91+
create: true
92+
# Telemetry
93+
- source:
94+
kind: ConfigMap
95+
name: service-values
96+
fieldPath: data.telemetry
97+
targets:
98+
- select:
99+
kind: OpenStackControlPlane
100+
fieldPaths:
101+
- spec.telemetry
102+
options:
103+
create: true
104+
# Neutron ML2 config
105+
- source:
106+
kind: ConfigMap
107+
name: service-values
108+
fieldPath: data.neutron.customServiceConfig
109+
targets:
110+
- select:
111+
kind: OpenStackControlPlane
112+
fieldPaths:
113+
- spec.neutron.template.customServiceConfig
114+
options:
115+
create: true
116+
# Nova PCI passthrough / scheduler config
117+
- source:
118+
kind: ConfigMap
119+
name: service-values
120+
fieldPath: data.nova.apiServiceTemplate.customServiceConfig
121+
targets:
122+
- select:
123+
kind: OpenStackControlPlane
124+
fieldPaths:
125+
- spec.nova.template.apiServiceTemplate.customServiceConfig
126+
options:
127+
create: true
128+
- source:
129+
kind: ConfigMap
130+
name: service-values
131+
fieldPath: data.nova.cell0.conductorServiceTemplate.customServiceConfig
132+
targets:
133+
- select:
134+
kind: OpenStackControlPlane
135+
fieldPaths:
136+
- spec.nova.template.cellTemplates.cell0.conductorServiceTemplate.customServiceConfig
137+
options:
138+
create: true
139+
- source:
140+
kind: ConfigMap
141+
name: service-values
142+
fieldPath: data.nova.cell1.conductorServiceTemplate.customServiceConfig
143+
targets:
144+
- select:
145+
kind: OpenStackControlPlane
146+
fieldPaths:
147+
- spec.nova.template.cellTemplates.cell1.conductorServiceTemplate.customServiceConfig
148+
options:
149+
create: true
150+
- source:
151+
kind: ConfigMap
152+
name: service-values
153+
fieldPath: data.nova.schedulerServiceTemplate.customServiceConfig
154+
targets:
155+
- select:
156+
kind: OpenStackControlPlane
157+
fieldPaths:
158+
- spec.nova.template.schedulerServiceTemplate.customServiceConfig
159+
options:
160+
create: true

‎dt/nova/nova05epsilon/edpm-post-ceph/nodeset/kustomization.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@ transformers:
1818
create: true
1919
2020
components:
21-
- ../../control-plane/
2221
- ../../../../../lib/dataplane/nodeset
2322

2423
resources:

‎dt/nova/nova05epsilon/kustomization.yaml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -43,17 +43,6 @@ replacements:
4343
- spec.ovn.template.ovnController.nicMappings
4444
options:
4545
create: true
46-
- source:
47-
kind: ConfigMap
48-
name: service-values
49-
fieldPath: data.glance.customServiceConfig
50-
targets:
51-
- select:
52-
kind: OpenStackControlPlane
53-
fieldPaths:
54-
- spec.glance.template.customServiceConfig
55-
options:
56-
create: true
5746
- source:
5847
kind: ConfigMap
5948
name: service-values
@@ -131,14 +120,3 @@ replacements:
131120
- spec.nova.template.schedulerServiceTemplate.customServiceConfig
132121
options:
133122
create: true
134-
- source:
135-
kind: ConfigMap
136-
name: service-values
137-
fieldPath: data.extraMounts
138-
targets:
139-
- select:
140-
kind: OpenStackControlPlane
141-
fieldPaths:
142-
- spec.extraMounts
143-
options:
144-
create: true

‎examples/dt/nova/nova05epsilon/README.md‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,20 @@ This is a collection of CR templates that represent a Red Hat OpenStack Services
2828

2929
5. Between stages 3 and 4, _it is assumed that the user installs Ceph on the compute nodes._ OpenStack K8S CRDs do not provide a way to install Ceph via any sort of combination of CRs.
3030

31-
6. For CI automation, this DT uses `automation/vars/nova05epsilon.yaml` which maps the manual stages above to 9 granular automation steps (NNCP, networking, control-plane, DNS, baremetalhosts, pre-ceph nodeset, pre-ceph deployment, post-ceph nodeset, post-ceph deployment).
31+
6. In stage 5 the `control-plane-post-ceph` kustomization needs the same network values used in stage 2 to preserve endpoint IPs, service types, and DNS configuration. For manual deployment, copy your environment-customized `control-plane/networking/nncp/values.yaml` into `control-plane-post-ceph/network-values.yaml` and populate `control-plane-post-ceph/values.yaml` with base64-encoded Ceph keyring and config. In CI, `ci_gen_kustomize_values` generates `network-values.yaml` in-place using the common Jinja2 template with environment overlays. See [control-plane-post-ceph.md](control-plane-post-ceph.md) for details.
32+
33+
7. On SNO with a single EDPM compute (single-host CephHCI), the Ceph ingress service (haproxy/keepalived) is not deployed. The default Swift endpoint (`<vip>:8080`) is unreachable because no ingress fronts the RGW daemon. Instead, clients must reach RGW directly on the compute's storage IP at port 8082 (the `rgw_frontend_port` set in the Ceph RGW spec).
34+
For CI automation, set `cifmw_cephadm_rgw_port: 8082` and `cifmw_cephadm_rgw_vip: <compute_storage_ip>` in the scenario vars so that `cifmw_cephadm` creates the Keystone endpoint with the correct address.
35+
For manual deployment, after installing Ceph, update the Swift endpoints in Keystone to point at the RGW daemon directly:
36+
37+
```shell
38+
STORAGE_IP=<compute storage network IP>
39+
for ep_id in $(openstack endpoint list --service object-store -f value -c ID); do
40+
openstack endpoint set --url "http://${STORAGE_IP}:8082/swift/v1/AUTH_%(tenant_id)s" "$ep_id"
41+
done
42+
```
43+
44+
8. For CI automation, this DT uses `automation/vars/nova05epsilon.yaml` which maps the manual stages above to 10 granular automation steps (NNCP, networking, control-plane, DNS, baremetalhosts, pre-ceph nodeset, pre-ceph deployment, control-plane-post-ceph, post-ceph nodeset, post-ceph deployment).
3245

3346
## Host Configuration
3447

@@ -57,7 +70,9 @@ All stages must be executed in the order listed below. Everything is required un
5770
1. [Install the OpenStack K8S operators and their dependencies](../../../common/)
5871
2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
5972
3. [Configure and deploy the initial data plane to prepare for Ceph installation](dataplane-pre-ceph.md)
60-
4. [Update the control plane and finish deploying the data plane after Ceph has been installed](dataplane-post-ceph.md)
73+
4. Install Ceph on the compute nodes (without changing OpenStack CP CR)
74+
5. [Update the control plane with Ceph backend configuration](control-plane-post-ceph.md)
75+
6. [Finish deploying the data plane after Ceph has been installed](dataplane-post-ceph.md)
6176

6277
## Extending to a Full DCN Deployment
6378

‎examples/dt/nova/nova05epsilon/control-plane-post-ceph.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Update the control plane with Ceph backend configuration
2+
3+
## Assumptions
4+
5+
- The [pre-ceph data plane](dataplane-pre-ceph.md) has been deployed
6+
- Ceph has been installed on the compute nodes
7+
- The `ceph-conf-files` secret will be created during this step
8+
9+
## Initialize
10+
11+
Switch to the "openstack" namespace
12+
13+
```shell
14+
oc project openstack
15+
```
16+
17+
Change to the control-plane-post-ceph directory
18+
19+
```shell
20+
cd architecture/examples/dt/nova/nova05epsilon/control-plane-post-ceph
21+
```
22+
23+
## Prepare values files
24+
25+
The kustomization requires three values files:
26+
27+
**network-values.yaml** — Provides storageClass, bridgeName, endpoint
28+
annotations, and DNS options that the lib components need. This file
29+
must match the `network-values` used in the initial control-plane stage
30+
to preserve all endpoint IPs, service types, and DNS configuration.
31+
In CI, `ci_gen_kustomize_values` generates it from the automation vars
32+
(`src_file: network-values.yaml`). For manual deployment, copy the
33+
environment-customized file from the control-plane networking stage:
34+
35+
```shell
36+
cp ../control-plane/networking/nncp/values.yaml network-values.yaml
37+
```
38+
39+
Edit `network-values.yaml` and replace any remaining `CHANGEME`
40+
placeholders to match your environment (same values used in stage 2).
41+
42+
**values.yaml** — Ceph configuration for the `ceph-conf-files` secret.
43+
Replace the `CHANGEME` placeholders with base64-encoded Ceph keyring
44+
and config from your Ceph deployment:
45+
46+
```shell
47+
vi values.yaml
48+
# Set data.ceph_conf."ceph.client.openstack.keyring" to:
49+
# base64 -w0 /etc/ceph/ceph.client.openstack.keyring
50+
# Set data.ceph_conf."ceph.conf" to:
51+
# base64 -w0 /etc/ceph/ceph.conf
52+
```
53+
54+
**service-values.yaml** — Edit if you need to adjust the Glance RBD
55+
backend or Ceph extraMounts configuration:
56+
57+
```shell
58+
vi service-values.yaml
59+
```
60+
61+
## Update the control plane
62+
63+
Generate the control-plane-post-ceph CRs:
64+
65+
```shell
66+
kustomize build > control-plane-post-ceph.yaml
67+
```
68+
69+
Apply the CRs:
70+
71+
```shell
72+
oc apply -f control-plane-post-ceph.yaml
73+
```
74+
75+
Wait for the control plane to be ready:
76+
77+
```shell
78+
oc wait osctlplane controlplane --for condition=Ready --timeout=1200s
79+
```

‎examples/dt/nova/nova05epsilon/control-plane-post-ceph/kustomization.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
5+
components:
6+
- ../../../../../dt/nova/nova05epsilon/control-plane-post-ceph
7+
8+
resources:
9+
- network-values.yaml
10+
- values.yaml
11+
- service-values.yaml

0 commit comments

Comments
 (0)