Skip to content

Commit 8d18c89

Browse files
bogdandobogdando
committed
nova05: separate pre and post ceph OSCP configs
HCI uses edpm post ceph to also patch the OSCP which looks suboptimal. Separate this into a two steps: * make post-ceph nodeset layer handling EDPM resources only (NodeSet, Services, Secrets, ConfigMap) - remove OSCP layer resources * add control-plane-post-ceph layer for OSCP with Glance RBD + Ceph extraMounts and secret config. Instead of re-including lib/control-plane (which required network-values duplication) provide only required values for lib/control-plane/base. * update README.md stages list from 4 to 6 steps, automation step count from 9 to 10 * add note for defferred Ceph config for control-plane.md * add control-plane post-ceph doc for the dedicated Ceph control-plane update stage * remove control-plane update references for dataplane-post-ceph.md, add prerequisite for control-plane-post-ceph * add control-plane-post-ceph stage between pre-ceph deployment and post-ceph nodeset ini automation vars Signed-off-by: Bohdan Dobrelia <bdobreli@redhat.com>
1 parent 4885ba3 commit 8d18c89

15 files changed

Lines changed: 699 additions & 70 deletions

File tree

‎automation/vars/nova05epsilon.yaml‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -105,14 +105,25 @@ vas:
105105
type: playbook
106106
source: "../../hooks/playbooks/fix_swift_endpoint.yml"
107107
inventory: "${HOME}/ci-framework-data/artifacts/ceph_inventory.yml"
108-
109-
- name: edpm-nodeset-post-ceph
110-
path: examples/dt/nova/nova05epsilon
108+
- name: control-plane-post-ceph
109+
path: examples/dt/nova/nova05epsilon/control-plane-post-ceph
111110
wait_conditions:
112111
- >-
113112
oc -n openstack wait osctlplane controlplane
114113
--for condition=Ready
115114
--timeout=20m
115+
values:
116+
- name: network-values
117+
src_file: ../control-plane/networking/nncp/values.yaml
118+
- name: service-values
119+
src_file: service-values.yaml
120+
- name: edpm-nodeset-values-post-ceph
121+
src_file: values.yaml
122+
build_output: control-plane-post-ceph.yaml
123+
124+
- name: edpm-nodeset-post-ceph
125+
path: examples/dt/nova/nova05epsilon
126+
wait_conditions:
116127
- >-
117128
oc -n openstack wait osdpns
118129
gpu-computes-edpm

‎dt/nova/nova05epsilon/control-plane-post-ceph/ceph_secret.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
apiVersion: v1
3+
data:
4+
ceph.client.openstack.keyring: _ignored_
5+
ceph.conf: _ignored_
6+
kind: Secret
7+
metadata:
8+
name: ceph-conf-files
9+
namespace: openstack
10+
type: Opaque

‎dt/nova/nova05epsilon/control-plane-post-ceph/kustomization.yaml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1alpha1
3+
kind: Component
4+
5+
transformers:
6+
- |-
7+
apiVersion: builtin
8+
kind: NamespaceTransformer
9+
metadata:
10+
name: _ignored_
11+
namespace: openstack
12+
setRoleBindingSubjects: none
13+
unsetOnly: true
14+
fieldSpecs:
15+
- path: metadata/name
16+
kind: Namespace
17+
create: true
18+
19+
components:
20+
- ../../../../lib/control-plane/base
21+
- ../../../../lib/control-plane/service-endpoints
22+
- ../../../../lib/control-plane/dns
23+
- ../../../../lib/control-plane/storage
24+
- ../../../../lib/control-plane/ovn-bridge
25+
- ../../../../lib/control-plane/job-settings
26+
27+
resources:
28+
- ceph_secret.yaml
29+
30+
replacements:
31+
# Ceph config files -> Secret
32+
- source:
33+
kind: ConfigMap
34+
name: edpm-nodeset-values-post-ceph
35+
fieldPath: data.ceph_conf
36+
targets:
37+
- select:
38+
kind: Secret
39+
name: ceph-conf-files
40+
fieldPaths:
41+
- data
42+
options:
43+
create: true
44+
# Glance RBD backend
45+
- source:
46+
kind: ConfigMap
47+
name: service-values
48+
fieldPath: data.glance.customServiceConfig
49+
targets:
50+
- select:
51+
kind: OpenStackControlPlane
52+
fieldPaths:
53+
- spec.glance.template.customServiceConfig
54+
options:
55+
create: true
56+
- source:
57+
kind: ConfigMap
58+
name: service-values
59+
fieldPath: data.glance.default.replicas
60+
targets:
61+
- select:
62+
kind: OpenStackControlPlane
63+
fieldPaths:
64+
- spec.glance.template.glanceAPIs.default.replicas
65+
options:
66+
create: true
67+
- source:
68+
kind: ConfigMap
69+
name: service-values
70+
fieldPath: data.extraMounts
71+
targets:
72+
- select:
73+
kind: OpenStackControlPlane
74+
fieldPaths:
75+
- spec.extraMounts
76+
options:
77+
create: true

‎dt/nova/nova05epsilon/edpm-post-ceph/nodeset/kustomization.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@ transformers:
1818
create: true
1919
2020
components:
21-
- ../../control-plane/
2221
- ../../../../../lib/dataplane/nodeset
2322

2423
resources:

‎dt/nova/nova05epsilon/kustomization.yaml‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -43,17 +43,6 @@ replacements:
4343
- spec.ovn.template.ovnController.nicMappings
4444
options:
4545
create: true
46-
- source:
47-
kind: ConfigMap
48-
name: service-values
49-
fieldPath: data.glance.customServiceConfig
50-
targets:
51-
- select:
52-
kind: OpenStackControlPlane
53-
fieldPaths:
54-
- spec.glance.template.customServiceConfig
55-
options:
56-
create: true
5746
- source:
5847
kind: ConfigMap
5948
name: service-values
@@ -131,14 +120,3 @@ replacements:
131120
- spec.nova.template.schedulerServiceTemplate.customServiceConfig
132121
options:
133122
create: true
134-
- source:
135-
kind: ConfigMap
136-
name: service-values
137-
fieldPath: data.extraMounts
138-
targets:
139-
- select:
140-
kind: OpenStackControlPlane
141-
fieldPaths:
142-
- spec.extraMounts
143-
options:
144-
create: true

‎examples/dt/nova/nova05epsilon/README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,11 @@ This is a collection of CR templates that represent a Red Hat OpenStack Services
2626

2727
4. In stages 3 and 4 [kustomize](https://kustomize.io/) is used to generate the dataplane CRs dynamically. The `edpm-pre-ceph/nodeset/values.yaml`, `values.yaml` and `service-values.yaml` files must be updated to fit your environment. kustomize version 5 or newer required.
2828

29+
5. In stage 5 the `control-plane-post-ceph` kustomization reuses network values from stage 2. For manual deployment, copy `control-plane/networking/nncp/values.yaml` into `control-plane-post-ceph/network-values.yaml` and populate `control-plane-post-ceph/values.yaml` with base64-encoded Ceph keyring and config. In CI, `ci_gen_kustomize_values` handles this automatically. See [control-plane-post-ceph.md](control-plane-post-ceph.md) for details.
30+
2931
5. Between stages 3 and 4, _it is assumed that the user installs Ceph on the compute nodes._ OpenStack K8S CRDs do not provide a way to install Ceph via any sort of combination of CRs.
3032

31-
6. For CI automation, this DT uses `automation/vars/nova05epsilon.yaml` which maps the manual stages above to 9 granular automation steps (NNCP, networking, control-plane, DNS, baremetalhosts, pre-ceph nodeset, pre-ceph deployment, post-ceph nodeset, post-ceph deployment).
33+
6. For CI automation, this DT uses `automation/vars/nova05epsilon.yaml` which maps the manual stages above to 10 granular automation steps (NNCP, networking, control-plane, DNS, baremetalhosts, pre-ceph nodeset, pre-ceph deployment, control-plane-post-ceph, post-ceph nodeset, post-ceph deployment).
3234

3335
7. On SNO with a single EDPM compute (single-host CephHCI), the Ceph ingress service (haproxy/keepalived) is not deployed. The post-ceph hook detects this and corrects the Keystone Swift endpoint. Pods that need to reach RGW (e.g. Tempest) must attach the `storage` and `ctlplane` NetworkAttachmentDefinitions.
3436

@@ -59,7 +61,9 @@ All stages must be executed in the order listed below. Everything is required un
5961
1. [Install the OpenStack K8S operators and their dependencies](../../../common/)
6062
2. [Configuring networking and deploy the OpenStack control plane](control-plane.md)
6163
3. [Configure and deploy the initial data plane to prepare for Ceph installation](dataplane-pre-ceph.md)
62-
4. [Update the control plane and finish deploying the data plane after Ceph has been installed](dataplane-post-ceph.md)
64+
4. Install Ceph on the compute nodes (without changing OpenStack CP CR)
65+
5. [Update the control plane with Ceph backend configuration](control-plane-post-ceph.md)
66+
6. [Finish deploying the data plane after Ceph has been installed](dataplane-post-ceph.md)
6367

6468
## Extending to a Full DCN Deployment
6569

‎examples/dt/nova/nova05epsilon/control-plane-post-ceph.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Update the control plane with Ceph backend configuration
2+
3+
## Assumptions
4+
5+
- The [pre-ceph data plane](dataplane-pre-ceph.md) has been deployed
6+
- Ceph has been installed on the compute nodes
7+
- The `ceph-conf-files` secret will be created during this step
8+
9+
## Initialize
10+
11+
Switch to the "openstack" namespace
12+
13+
```shell
14+
oc project openstack
15+
```
16+
17+
Change to the control-plane-post-ceph directory
18+
19+
```shell
20+
cd architecture/examples/dt/nova/nova05epsilon/control-plane-post-ceph
21+
```
22+
23+
## Prepare values files
24+
25+
The kustomization requires three values files:
26+
27+
**network-values.yaml** — Copy from the control-plane networking stage.
28+
This provides storageClass, bridgeName, endpoint annotations, and DNS
29+
options that the lib components need. In CI this is injected by
30+
`ci_gen_kustomize_values`; for manual deployment, copy the file:
31+
32+
```shell
33+
cp ../control-plane/networking/nncp/values.yaml network-values.yaml
34+
```
35+
36+
**values.yaml** — Ceph configuration for the `ceph-conf-files` secret.
37+
Replace the `CHANGEME` placeholders with base64-encoded Ceph keyring
38+
and config from your Ceph deployment:
39+
40+
```shell
41+
vi values.yaml
42+
# Set data.ceph_conf."ceph.client.openstack.keyring" to:
43+
# base64 -w0 /etc/ceph/ceph.client.openstack.keyring
44+
# Set data.ceph_conf."ceph.conf" to:
45+
# base64 -w0 /etc/ceph/ceph.conf
46+
```
47+
48+
**service-values.yaml** — Edit if you need to adjust the Glance RBD
49+
backend or Ceph extraMounts configuration:
50+
51+
```shell
52+
vi service-values.yaml
53+
```
54+
55+
## Update the control plane
56+
57+
Generate the control-plane-post-ceph CRs:
58+
59+
```shell
60+
kustomize build > control-plane-post-ceph.yaml
61+
```
62+
63+
Apply the CRs:
64+
65+
```shell
66+
oc apply -f control-plane-post-ceph.yaml
67+
```
68+
69+
Wait for the control plane to be ready:
70+
71+
```shell
72+
oc wait osctlplane controlplane --for condition=Ready --timeout=1200s
73+
```

‎examples/dt/nova/nova05epsilon/control-plane-post-ceph/kustomization.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
apiVersion: kustomize.config.k8s.io/v1beta1
3+
kind: Kustomization
4+
5+
components:
6+
- ../../../../../dt/nova/nova05epsilon/control-plane-post-ceph
7+
8+
resources:
9+
- network-values.yaml
10+
- values.yaml
11+
- service-values.yaml

0 commit comments

Comments
 (0)