Skip to content

Commit 19089a8

Browse files
bogdandobogdando
committed
nova05: separate pre and post ceph OSCP configs
HCI uses edpm post ceph to also patch the OSCP which looks suboptimal. Separate this into a two steps: * make post-ceph nodeset layer handling EDPM resources only (NodeSet, Services, Secrets, ConfigMap) - remove OSCP layer resources * add control-plane-post-ceph layer for OSCP with Glance RBD + Ceph extraMounts and secret config. Instead of re-including lib/control-plane (which required network-values duplication) provide only required values for lib/control-plane/base. * update README.md stages list from 4 to 6 steps, automation step count from 9 to 10 * add note for defferred Ceph config for control-plane.md * add control-plane post-ceph doc for the dedicated Ceph control-plane update stage * remove control-plane update references for dataplane-post-ceph.md, add prerequisite for control-plane-post-ceph * add control-plane-post-ceph stage between pre-ceph deployment and post-ceph nodeset ini automation vars Signed-off-by: Bohdan Dobrelia <bdobreli@redhat.com>
1 parent c1a6e5c commit 19089a8

15 files changed

Lines changed: 837 additions & 70 deletions

File tree

‎automation/vars/nova05epsilon.yaml‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,14 +101,25 @@ vas:
101101
type: playbook
102102
source: "../../hooks/playbooks/ceph.yml"
103103
inventory: "${HOME}/ci-framework-data/artifacts/ceph_inventory.yml"
104-
105-
- name: edpm-nodeset-post-ceph
106-
path: examples/dt/nova/nova05epsilon
104+
- name: control-plane-post-ceph
105+
path: examples/dt/nova/nova05epsilon/control-plane-post-ceph
107106
wait_conditions:
108107
- >-
109108
oc -n openstack wait osctlplane controlplane
110109
--for condition=Ready
111110
--timeout=20m
111+
values:
112+
- name: network-values
113+
src_file: ../control-plane/networking/nncp/values.yaml
114+
- name: service-values
115+
src_file: service-values.yaml
116+
- name: edpm-nodeset-values-post-ceph
117+
src_file: values.yaml
118+
build_output: control-plane-post-ceph.yaml
119+
120+
- name: edpm-nodeset-post-ceph
121+
path: examples/dt/nova/nova05epsilon
122+
wait_conditions:
112123
- >-
113124
oc -n openstack wait osdpns
114125
gpu-computes-edpm

‎dt/nova/nova05epsilon/control-plane-post-ceph/ceph_secret.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
data:
4+
ceph.client.openstack.keyring: _ignored_
5+
ceph.conf: _ignored_
6+
kind: Secret
7+
metadata:
8+
name: ceph-conf-files
9+
namespace: openstack
10+
type: Opaque

‎dt/nova/nova05epsilon/control-plane-post-ceph/kustomization.yaml‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1alpha1
3+
kind: Component
4+
5+
transformers:
6+
- |-
7+
apiVersion: builtin
8+
kind: NamespaceTransformer
9+
metadata:
10+
name: _ignored_
11+
namespace: openstack
12+
setRoleBindingSubjects: none
13+
unsetOnly: true
14+
fieldSpecs:
15+
- path: metadata/name
16+
kind: Namespace
17+
create: true
18+
19+
components:
20+
- ../../../../lib/control-plane/base
21+
- ../../../../lib/control-plane/service-endpoints
22+
- ../../../../lib/control-plane/dns
23+
- ../../../../lib/control-plane/storage
24+
- ../../../../lib/control-plane/ovn-bridge
25+
- ../../../../lib/control-plane/job-settings
26+
- ../../../../lib/control-plane/tls
27+
- ../../../../lib/control-plane/messaging-bus
28+
29+
resources:
30+
- ceph_secret.yaml
31+
32+
replacements:
33+
# Ceph config files -> Secret
34+
- source:
35+
kind: ConfigMap
36+
name: edpm-nodeset-values-post-ceph
37+
fieldPath: data.ceph_conf
38+
targets:
39+
- select:
40+
kind: Secret
41+
name: ceph-conf-files
42+
fieldPaths:
43+
- data
44+
options:
45+
create: true
46+
# Glance RBD backend
47+
- source:
48+
kind: ConfigMap
49+
name: service-values
50+
fieldPath: data.glance.customServiceConfig
51+
targets:
52+
- select:
53+
kind: OpenStackControlPlane
54+
fieldPaths:
55+
- spec.glance.template.customServiceConfig
56+
options:
57+
create: true
58+
- source:
59+
kind: ConfigMap
60+
name: service-values
61+
fieldPath: data.glance.default.replicas
62+
targets:
63+
- select:
64+
kind: OpenStackControlPlane
65+
fieldPaths:
66+
- spec.glance.template.glanceAPIs.default.replicas
67+
options:
68+
create: true
69+
- source:
70+
kind: ConfigMap
71+
name: service-values
72+
fieldPath: data.extraMounts
73+
targets:
74+
- select:
75+
kind: OpenStackControlPlane
76+
fieldPaths:
77+
- spec.extraMounts
78+
options:
79+
create: true
80+
# Swift
81+
- source:
82+
kind: ConfigMap
83+
name: service-values
84+
fieldPath: data.swift.enabled
85+
targets:
86+
- select:
87+
kind: OpenStackControlPlane
88+
fieldPaths:
89+
- spec.swift.enabled
90+
options:
91+
create: true
92+
# Telemetry
93+
- source:
94+
kind: ConfigMap
95+
name: service-values
96+
fieldPath: data.telemetry
97+
targets:
98+
- select:
99+
kind: OpenStackControlPlane
100+
fieldPaths:
101+
- spec.telemetry
102+
options:
103+
create: true
104+
# Neutron ML2 config
105+
- source:
106+
kind: ConfigMap
107+
name: service-values
108+
fieldPath: data.neutron.customServiceConfig
109+
targets:
110+
- select:
111+
kind: OpenStackControlPlane
112+
fieldPaths:
113+
- spec.neutron.template.customServiceConfig
114+
options:
115+
create: true
116+
# Nova PCI passthrough / scheduler config
117+
- source:
118+
kind: ConfigMap
119+
name: service-values
120+
fieldPath: data.nova.apiServiceTemplate.customServiceConfig
121+
targets:
122+
- select:
123+
kind: OpenStackControlPlane
124+
fieldPaths:
125+
- spec.nova.template.apiServiceTemplate.customServiceConfig
126+
options:
127+
create: true
128+
- source:
129+
kind: ConfigMap
130+
name: service-values
131+
fieldPath: data.nova.cell0.conductorServiceTemplate.customServiceConfig
132+
targets:
133+
- select:
134+
kind: OpenStackControlPlane
135+
fieldPaths:
136+
- spec.nova.template.cellTemplates.cell0.conductorServiceTemplate.customServiceConfig
137+
options:
138+
create: true
139+
- source:
140+
kind: ConfigMap
141+
name: service-values
142+
fieldPath: data.nova.cell1.conductorServiceTemplate.customServiceConfig
143+
targets:
144+
- select:
145+
kind: OpenStackControlPlane
146+
fieldPaths:
147+
- spec.nova.template.cellTemplates.cell1.conductorServiceTemplate.customServiceConfig
148+
options:
149+
create: true
150+
- source:
151+
kind: ConfigMap
152+
name: service-values
153+
fieldPath: data.nova.schedulerServiceTemplate.customServiceConfig
154+
targets:
155+
- select:
156+
kind: OpenStackControlPlane
157+
fieldPaths:
158+
- spec.nova.template.schedulerServiceTemplate.customServiceConfig
159+
options:
160+
create: true

‎dt/nova/nova05epsilon/edpm-post-ceph/nodeset/kustomization.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@ transformers:
1818
create: true
1919
2020
components:
21-
- ../../control-plane/
2221
- ../../../../../lib/dataplane/nodeset
2322

2423
resources:

‎dt/nova/nova05epsilon/kustomization.yaml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -43,17 +43,6 @@ replacements:
4343
- spec.ovn.template.ovnController.nicMappings
4444
options:
4545
create: true
46-
- source:
47-
kind: ConfigMap
48-
name: service-values
49-
fieldPath: data.glance.customServiceConfig
50-
targets:
51-
- select:
52-
kind: OpenStackControlPlane
53-
fieldPaths:
54-
- spec.glance.template.customServiceConfig
55-
options:
56-
create: true
5746
- source:
5847
kind: ConfigMap
5948
name: service-values
@@ -131,14 +120,3 @@ replacements:
131120
- spec.nova.template.schedulerServiceTemplate.customServiceConfig
132121
options:
133122
create: true
134-
- source:
135-
kind: ConfigMap
136-
name: service-values
137-
fieldPath: data.extraMounts
138-
targets:
139-
- select:
140-
kind: OpenStackControlPlane
141-
fieldPaths:
142-
- spec.extraMounts
143-
options:
144-
create: true

‎examples/dt/nova/nova05epsilon/README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,11 @@ This is a collection of CR templates that represent a Red Hat OpenStack Services
2626

2727
4. In stages 3 and 4 [kustomize](https://kustomize.io/) is used to generate the dataplane CRs dynamically. The `edpm-pre-ceph/nodeset/values.yaml`, `values.yaml` and `service-values.yaml` files must be updated to fit your environment. kustomize version 5 or newer required.
2828

29+
5. In stage 5 the `control-plane-post-ceph` kustomization reuses network values from stage 2. For manual deployment, copy `control-plane/networking/nncp/values.yaml` into `control-plane-post-ceph/network-values.yaml` and populate `control-plane-post-ceph/values.yaml` with base64-encoded Ceph keyring and config. In CI, `ci_gen_kustomize_values` handles this automatically. See [control-plane-post-ceph.md](control-plane-post-ceph.md) for details.
30+
2931
5. Between stages 3 and 4, _it is assumed that the user installs Ceph on the compute nodes._ OpenStack K8S CRDs do not provide a way to install Ceph via any sort of combination of CRs.
3032

31-
6. For CI automation, this DT uses `automation/vars/nova05epsilon.yaml` which maps the manual stages above to 9 granular automation steps (NNCP, networking, control-plane, DNS, baremetalhosts, pre-ceph nodeset, pre-ceph deployment, post-ceph nodeset, post-ceph deployment).
33+
6. For CI automation, this DT uses `automation/vars/nova05epsilon.yaml` which maps the manual stages above to 10 granular automation steps (NNCP, networking, control-plane, DNS, baremetalhosts, pre-ceph nodeset, pre-ceph deployment, control-plane-post-ceph, post-ceph nodeset, post-ceph deployment).
3234

3335
## Host Configuration
3436

@@ -57,7 +59,9 @@ All stages must be executed in the order listed below. Everything is required un
5759
1. [Install the OpenStack K8S operators and their dependencies](../../../common/)
5860
2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
5961
3. [Configure and deploy the initial data plane to prepare for Ceph installation](dataplane-pre-ceph.md)
60-
4. [Update the control plane and finish deploying the data plane after Ceph has been installed](dataplane-post-ceph.md)
62+
4. Install Ceph on the compute nodes (without changing OpenStack CP CR)
63+
5. [Update the control plane with Ceph backend configuration](control-plane-post-ceph.md)
64+
6. [Finish deploying the data plane after Ceph has been installed](dataplane-post-ceph.md)
6165

6266
## Extending to a Full DCN Deployment
6367

‎examples/dt/nova/nova05epsilon/control-plane-post-ceph.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Update the control plane with Ceph backend configuration
2+
3+
## Assumptions
4+
5+
- The [pre-ceph data plane](dataplane-pre-ceph.md) has been deployed
6+
- Ceph has been installed on the compute nodes
7+
- The `ceph-conf-files` secret will be created during this step
8+
9+
## Initialize
10+
11+
Switch to the "openstack" namespace
12+
13+
```shell
14+
oc project openstack
15+
```
16+
17+
Change to the control-plane-post-ceph directory
18+
19+
```shell
20+
cd architecture/examples/dt/nova/nova05epsilon/control-plane-post-ceph
21+
```
22+
23+
## Prepare values files
24+
25+
The kustomization requires three values files:
26+
27+
**network-values.yaml** — Copy from the control-plane networking stage.
28+
This provides storageClass, bridgeName, endpoint annotations, and DNS
29+
options that the lib components need. In CI this is injected by
30+
`ci_gen_kustomize_values`; for manual deployment, copy the file:
31+
32+
```shell
33+
cp ../control-plane/networking/nncp/values.yaml network-values.yaml
34+
```
35+
36+
**values.yaml** — Ceph configuration for the `ceph-conf-files` secret.
37+
Replace the `CHANGEME` placeholders with base64-encoded Ceph keyring
38+
and config from your Ceph deployment:
39+
40+
```shell
41+
vi values.yaml
42+
# Set data.ceph_conf."ceph.client.openstack.keyring" to:
43+
# base64 -w0 /etc/ceph/ceph.client.openstack.keyring
44+
# Set data.ceph_conf."ceph.conf" to:
45+
# base64 -w0 /etc/ceph/ceph.conf
46+
```
47+
48+
**service-values.yaml** — Edit if you need to adjust the Glance RBD
49+
backend or Ceph extraMounts configuration:
50+
51+
```shell
52+
vi service-values.yaml
53+
```
54+
55+
## Update the control plane
56+
57+
Generate the control-plane-post-ceph CRs:
58+
59+
```shell
60+
kustomize build > control-plane-post-ceph.yaml
61+
```
62+
63+
Apply the CRs:
64+
65+
```shell
66+
oc apply -f control-plane-post-ceph.yaml
67+
```
68+
69+
Wait for the control plane to be ready:
70+
71+
```shell
72+
oc wait osctlplane controlplane --for condition=Ready --timeout=1200s
73+
```

‎examples/dt/nova/nova05epsilon/control-plane-post-ceph/kustomization.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
5+
components:
6+
- ../../../../../dt/nova/nova05epsilon/control-plane-post-ceph
7+
8+
resources:
9+
- network-values.yaml
10+
- values.yaml
11+
- service-values.yaml

0 commit comments

Comments
 (0)