Skip to content

Commit 3c627c8

Browse files
abhishek-sa1abhishek-sa1
committed
feat(build_manager): complete credential refactoring with validation integration
- Add Build Manager to credential validation flow (credential_utility/roles/validation/vars/main.yml) - Add build_manager credential paths to common role vars (utils/roles/common/vars/main.yml) - Add include_build_manager_credentials.yml task file for decrypt→load→re-encrypt flow - Add build_manager and build_image tags to validate_config.yml - Add build_manager to input_file_inventory in config.py (validates storage_config + software_config) - Fix standalone credential loading in prepare_build_manager.yml to load build_manager_credentials.yml - Update REFACTORING_SUMMARY.md with validation/common flow sections - Remove build_credentials.yml from input/ (credentials loaded dynamically) - Overwrite git-mv'd files with new idempotent content (minio, registry tasks + templates) Total: 93 files changed, +1731/-1023 lines
1 parent 99a5f7f commit 3c627c8

93 files changed

Lines changed: 1731 additions & 1023 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎src/build_manager/REFACTORING_SUMMARY.md‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Build Manager Credential & Directory Refactoring — Summary
2+
3+
## Overview
4+
5+
This refactoring separates **S3/MinIO credentials** and **MinIO/Registry deployment** from the
6+
`prepare_oim` domain and moves them exclusively to the `build_manager` domain. It also introduces
7+
dedicated input/output directory variables (`input_project_dir`, `output_project_dir`) so that
8+
`build_manager` and all downstream playbooks use variable-driven paths instead of hardcoded ones.
9+
10+
---
11+
12+
## What Changed
13+
14+
### 1. Credential Ownership Transfer
15+
16+
| Credential | Before (prepare_oim) | After (build_manager) |
17+
|---------------------|----------------------------------|--------------------------------------------|
18+
| `s3_secret_key` | In `omnia_config_credentials.yml`| In `build_manager_credentials.yml` only |
19+
| `s3_access_id` | In `omnia_config_credentials.yml`| In `build_manager_credentials.yml` only |
20+
| `provision_password`| In `omnia_config_credentials.yml`| In `build_manager_credentials.yml` (mandatory) |
21+
| `pulp_password` | In `omnia_config_credentials.yml`| Stays in `omnia_config_credentials.yml` |
22+
23+
**Key change:** `prepare_oim` no longer prompts for S3 credentials. Running `omnia.sh --tags prepare_oim`
24+
will only ask for `pulp_password` and optional Docker credentials.
25+
26+
#### Files changed:
27+
- `credential_utility/roles/update_config/vars/main.yml` — removed `s3_secret_key` and `s3_access_id`
28+
from `prepare_oim` tag; added `provision_password` as mandatory in `build_manager` tag.
29+
- `credential_utility/roles/create_config/templates/omnia_credential.j2` — removed S3 fields.
30+
- `credential_utility/roles/create_config/templates/build_manager_credential.j2` — added
31+
`provision_password`.
32+
- `credential_utility/roles/create_config/vars/main.yml` — Build Manager credential file entry
33+
at `{{ input_project_dir }}/build_manager_credentials.yml`.
34+
35+
### 2. MinIO/Registry Deployment Moved to build_manager
36+
37+
The following files were **git rm**'d from `prepare_oim` (they now live in `build_manager`):
38+
39+
```
40+
DELETED prepare_oim/roles/deploy_containers/openchami/tasks/configs/minio.yml
41+
DELETED prepare_oim/roles/deploy_containers/openchami/tasks/configs/registry.yml
42+
DELETED prepare_oim/roles/deploy_containers/openchami/tasks/configs/s3_bucket.yml
43+
DELETED prepare_oim/roles/deploy_containers/openchami/tasks/configs/policy_update.yml
44+
DELETED prepare_oim/roles/deploy_containers/openchami/templates/minio/minio.service.j2
45+
DELETED prepare_oim/roles/deploy_containers/openchami/templates/registry/registry.service.j2
46+
DELETED prepare_oim/roles/deploy_containers/openchami/templates/s3/s3cfg.j2
47+
DELETED prepare_oim/roles/deploy_containers/openchami/templates/s3/s3-public-read-boot.json.j2
48+
DELETED prepare_oim/roles/deploy_containers/openchami/templates/s3/s3-public-read-efi.json.j2
49+
```
50+
51+
The build_manager domain now handles:
52+
- MinIO container deployment (`build_manager/roles/deploy_minio/`)
53+
- Registry container deployment
54+
- S3 bucket creation and ACL policies
55+
56+
#### Files changed:
57+
- `prepare_oim/roles/deploy_containers/openchami/tasks/configs/main.yml` — removed imports of
58+
`minio.yml`, `registry.yml`, `s3_bucket.yml`, `policy_update.yml`.
59+
- `prepare_oim/roles/deploy_containers/openchami/tasks/deploy_openchami.yml` — removed
60+
`s3_access_id` / `s3_secret_key` fact-setting block (lines 156-160).
61+
62+
### 3. Input/Output Directory Variables
63+
64+
#### New variables:
65+
- `omnia_output_dir` — `/opt/omnia/output` (in `include_input_dir/vars/main.yml`)
66+
- `output_project_dir` — `{{ omnia_output_dir }}/{{ project_name }}` (set as fact in
67+
`include_input_dir/tasks/main.yml`, parallel to `input_project_dir`)
68+
69+
#### How it works:
70+
1. `omnia.sh` creates the output directory: `mkdir -p /opt/omnia/output/project_default`
71+
2. `omnia.sh` copies build_manager input files to `{{ input_project_dir }}/build_manager/`
72+
(no credential files — those are loaded dynamically by the credential utility)
73+
3. `include_input_dir` role sets both `input_project_dir` and `output_project_dir` as facts
74+
4. `build_manager.yml` uses `output_project_dir` for all output paths
75+
76+
#### Files changed:
77+
- `playbooks/utils/roles/include_input_dir/vars/main.yml` — added `omnia_output_dir`.
78+
- `playbooks/utils/roles/include_input_dir/tasks/main.yml` — added `output_project_dir` fact.
79+
- `main/omnia.sh` (`post_setup_config`) — creates output dir and copies build_manager inputs.
80+
81+
### 4. build_manager.yml Playbook Updates
82+
83+
- **Step 3** (repo_manager pre-check): `_repo_status_path` default now uses `output_project_dir`.
84+
- **Step 8** (write build_status): Uses `_output_base` / `_output_domain` from `output_project_dir`.
85+
- **write_build_status.yml** role: Writes to `{{ output_project_dir }}/build_manager/build_status.yml`.
86+
- Removed EFI fields and metadata (omnia_version, timestamp) from `build_status_data` output.
87+
- `build_manager_config.yml`: `repo_manager_output` default updated to use `output_project_dir`.
88+
89+
### 5. Upgrade & Rollback Compatibility
90+
91+
#### Upgrade (`restore_omnia_config_credentials.yml`):
92+
- Removed S3 creds from main `set_fact` (no longer in omnia_config_credentials.yml).
93+
- **Migration block preserved**: If upgrading from an older version that stored S3 creds in
94+
`omnia_config_credentials.yml`, they are automatically migrated to `build_manager_credentials.yml`.
95+
96+
#### Rollback (`load_rollback_credentials.yml`):
97+
- Removed S3 creds from main `set_fact`.
98+
- Loads S3 + `provision_password` from `build_manager_credentials.yml` when available.
99+
- Fallback: reads S3 from old `omnia_config_credentials.yml` for backward compat with older backups.
100+
101+
#### Files changed:
102+
- `upgrade/roles/import_input_parameters/templates/omnia_config_credentials.yml.j2` — removed S3.
103+
- `upgrade/roles/import_input_parameters/templates/build_manager_credentials.yml.j2` — added
104+
`provision_password`.
105+
- `upgrade/roles/import_input_parameters/tasks/restore_omnia_config_credentials.yml` — removed S3
106+
from main set_fact; migration block handles old→new.
107+
- `rollback/playbooks/load_rollback_credentials.yml` — removed S3 from main set_fact; loads from
108+
build_manager_credentials.yml.
109+
110+
---
111+
112+
## How It Works (End-to-End Flow)
113+
114+
```
115+
┌─────────────────────────────────────────────────────────┐
116+
│ omnia.sh │
117+
│ 1. Creates /opt/omnia/output/project_default/ │
118+
│ 2. Copies build_manager input → input/project_default/ │
119+
│ build_manager/ (no credential files) │
120+
└────────────────────┬────────────────────────────────────┘
121+
│
122+
▼
123+
┌─────────────────────────────────────────────────────────┐
124+
│ credential_utility │
125+
│ Tag: build_manager │
126+
│ Prompts: s3_secret_key, s3_access_id*, provision_pass │
127+
│ Writes: build_manager_credentials.yml (vault-encrypted)│
128+
│ (* conditional: only if provider == 'powerscale') │
129+
│ │
130+
│ Tag: prepare_oim │
131+
│ Prompts: pulp_password, docker_username/password* │
132+
│ Writes: omnia_config_credentials.yml (NO S3 creds) │
133+
└────────────────────┬────────────────────────────────────┘
134+
│
135+
▼
136+
┌─────────────────────────────────────────────────────────┐
137+
│ include_input_dir role │
138+
│ Sets facts: │
139+
│ input_project_dir = /opt/omnia/input/project_default│
140+
│ output_project_dir = /opt/omnia/output/project_default│
141+
└────────────────────┬────────────────────────────────────┘
142+
│
143+
▼
144+
┌─────────────────────────────────────────────────────────┐
145+
│ build_manager.yml │
146+
│ 1. Loads build_manager_credentials.yml (S3 + provision)│
147+
│ 2. Reads repo_status.yml from output_project_dir │
148+
│ 3. Deploys MinIO + Registry (via deploy_minio role) │
149+
│ 4. Builds images (x86_64 local, aarch64 via SSH) │
150+
│ 5. Writes build_status.yml → output_project_dir/ │
151+
│ build_manager/ │
152+
└─────────────────────────────────────────────────────────┘
153+
```
154+
155+
### Credential File Locations
156+
157+
```
158+
/opt/omnia/input/project_default/
159+
├── omnia_config_credentials.yml ← pulp, bmc, docker, slurm, etc.
160+
├── .omnia_config_credentials_key
161+
├── build_manager_credentials.yml ← s3_access_id, s3_secret_key, provision_password
162+
├── .build_manager_credentials_key
163+
├── build_manager/ ← build_manager input configs (NOT creds)
164+
│ ├── build_manager_config.yml
165+
│ └── storage_config.yml
166+
└── ...other input files...
167+
168+
/opt/omnia/output/project_default/
169+
├── repo_manager/
170+
│ └── repo_status.yml ← consumed by build_manager Step 3
171+
└── build_manager/
172+
└── build_status.yml ← written by build_manager Step 8
173+
```
174+
175+
### 6. Input Validation & Common Flow Updates
176+
177+
The following files were added or updated to integrate build_manager into the shared validation
178+
and common credential-loading flows:
179+
180+
#### Credential Validation (`credential_utility/roles/validation/vars/main.yml`):
181+
- Added **Build Manager** entry to `credential_files` list so `build_manager_credentials.yml`
182+
is validated for existence during the credential creation flow.
183+
184+
#### Common Role (`playbooks/utils/roles/common/`):
185+
- `vars/main.yml` — Added `build_manager_credentials_filename`, `build_manager_credentials_vault_path`,
186+
`build_manager_credentials_syntax_fail_msg`, `build_manager_config_filename`, and
187+
`build_manager_config_syntax_fail_msg` variables.
188+
- `tasks/main.yml` — Added include of `include_build_manager_credentials.yml` with
189+
`build_manager_credentials_config` tag.
190+
- `tasks/include_build_manager_credentials.yml` — **New file.** Decrypt → load → re-encrypt
191+
flow for `build_manager_credentials.yml` (mirrors `include_omnia_config_credentials.yml`).
192+
193+
#### Input Validation Framework (`input_validation/`):
194+
- `validate_config.yml` — Added `build_image` and `build_manager` to common tag anchor list.
195+
- `common/library/module_utils/input_validation/common_utils/config.py`:
196+
- Added `build_manager` tag to `input_file_inventory` → validates `storage_config.yml` +
197+
`software_config.json`.
198+
- Added `build_manager_credentials.yml` to `get_vault_password()` mapping.
199+
200+
#### Standalone Playbook Fix (`playbooks/prepare_build_manager.yml`):
201+
- Standalone credential loading now reads `build_manager_credentials.yml` instead of
202+
`omnia_config_credentials.yml` (S3 creds moved out of omnia_config).
203+
204+
### 7. Removed Files
205+
206+
- `build_manager/input/build_credentials.yml` — Removed. This was a reference file that
207+
documented credential fields. Credentials are loaded dynamically via the credential utility,
208+
not from files in the input directory.
209+
210+
---
211+
212+
## Backward Compatibility
213+
214+
- **No breaking changes** for users who don't use build_manager.
215+
- **prepare_oim** no longer prompts for S3 creds — one fewer password to enter.
216+
- **Upgrade from older versions**: S3 creds in old `omnia_config_credentials.yml` are automatically
217+
migrated to `build_manager_credentials.yml`.
218+
- **Rollback**: Falls back to reading S3 from old credential file if
219+
`build_manager_credentials.yml` doesn't exist yet.

src/playbooks/build_image_aarch64/ansible.cfg renamed to src/build_manager/ansible.cfg

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[defaults]
2-
log_path = /opt/omnia/log/core/playbooks/build_image_aarch64.log
2+
log_path = /opt/omnia/log/core/playbooks/build_manager.log
33
remote_tmp = /opt/omnia/tmp/.ansible/tmp/
44
host_key_checking = false
55
forks = 5
@@ -9,9 +9,10 @@ interpreter_python = /usr/bin/python3
99
deprecation_warnings = false
1010
show_task_path_on_failure = false
1111
stdout_callback = omnia_default
12-
callback_plugins = ../../common/callback_plugins
13-
library = ../../common/library/modules
14-
module_utils = ../../common/library/module_utils
12+
roles_path = roles:../playbooks/input_validation/roles
13+
callback_plugins = ../common/callback_plugins
14+
library = ../common/library/modules
15+
module_utils = ../common/library/module_utils
1516

1617
[persistent_connection]
1718
command_timeout = 180

0 commit comments

Comments
 (0)