Skip to content

Commit 7f0ab18

Browse files
nicholaskuechlernicholaskuechler
committed
feat(ironic): Adds ironic runbook for performing BMC maintenances.
1 parent 2704035 commit 7f0ab18

9 files changed

Lines changed: 378 additions & 55 deletions

File tree

‎components/ironic/runbook-crd/README.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,52 @@ spec:
9090
| `runbook_disk_cleaning.yaml` | Node Reuse | Secure disk erasure |
9191
| `runbook_gpu_node_setup.yaml` | ML/AI | GPU node configuration |
9292

93+
## Running a Runbook
94+
95+
Once the operator syncs the CRD into Ironic, you execute a runbook by
96+
transitioning a node into the `clean` provisioning state with the runbook
97+
specified. The node must be in `manageable` state before you can trigger
98+
cleaning.
99+
100+
### OpenStack CLI
101+
102+
```bash
103+
# Run a runbook by name
104+
openstack baremetal node clean <node-uuid> --runbook CUSTOM_BMC_MAINTENANCE
105+
106+
# Check node state while cleaning runs
107+
openstack baremetal node show <node-uuid> -f value -c provision_state
108+
```
109+
110+
### Python SDK
111+
112+
```python
113+
from understack_workflows.ironic_node import transition
114+
115+
# node must already be in manageable state
116+
transition(
117+
node,
118+
"clean",
119+
expected_state="manageable",
120+
runbook=runbook_uuid,
121+
)
122+
```
123+
124+
The `transition` helper calls `set_node_provision_state` and waits for the
125+
node to return to `manageable` once all steps complete.
126+
127+
### Trait-Based Automatic Execution
128+
129+
Runbooks can also be triggered automatically by matching node traits. Add the
130+
runbook name as a trait on the node:
131+
132+
```bash
133+
openstack baremetal node add trait <node-uuid> CUSTOM_BMC_MAINTENANCE
134+
```
135+
136+
Workflow code (e.g. `apply_firmware_updates` in `ironic_node.py`) can then
137+
discover matching traits and execute the corresponding runbooks in order.
138+
93139
## Support
94140

95141
- **Ironic Documentation**: https://docs.openstack.org/ironic/latest/

‎components/ironic/runbook-crd/bases/baremetal.ironicproject.org_runbooks.yaml‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,25 @@ spec:
4141
- steps
4242
properties:
4343
runbookName:
44-
description: 'RunbookName is the unique name of the runbook (REQUIRED). Must match trait naming convention, typically CUSTOM_*. This name is used to match runbooks to nodes via traits.'
44+
description: 'RunbookName is the unique name of the runbook (REQUIRED). From API microversion 1.112+, this is a logical identifier and can be any string of 1-255 characters. Node eligibility is determined by the traits field instead.'
4545
type: string
46-
pattern: '^CUSTOM_[A-Z0-9_]+$'
46+
pattern: '^[a-zA-Z0-9._-]+$'
4747
minLength: 1
4848
maxLength: 255
49+
description:
50+
description: 'Description is a human-readable description of the runbook (OPTIONAL). Consistent with other Ironic objects. Available from API microversion 1.112 onwards.'
51+
type: string
52+
nullable: true
53+
maxLength: 1000
54+
traits:
55+
description: 'Traits is a list of traits that determine which nodes are permitted to use this runbook (OPTIONAL). Decouples runbook eligibility from the runbook name. Each trait must follow the CUSTOM_* naming convention. Available from API microversion 1.112 onwards. Default: []'
56+
type: array
57+
default: []
58+
items:
59+
type: string
60+
pattern: '^CUSTOM_[A-Z0-9_]+$'
61+
minLength: 1
62+
maxLength: 255
4963
steps:
5064
description: 'Steps is an ordered list of operations to execute (REQUIRED). Minimum 1 step required.'
5165
type: array
@@ -164,8 +178,13 @@ spec:
164178
additionalPrinterColumns:
165179
- name: Runbook Name
166180
type: string
167-
description: The runbook name used for trait matching
181+
description: The runbook name
168182
jsonPath: .spec.runbookName
183+
- name: Description
184+
type: string
185+
description: Human-readable description of the runbook
186+
jsonPath: .spec.description
187+
priority: 1
169188
- name: Steps
170189
type: integer
171190
description: Number of steps in the runbook

‎components/ironic/runbook-crd/kustomization.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@ namespace: openstack
77
# Create namespace if it doesn't exist
88
resources:
99
- bases/baremetal.ironicproject.org_runbooks.yaml
10+
- runbooks/runbook_bmc_maintenance.yaml

‎components/ironic/runbook-crd/runbooks/runbook_bmc_maintenance.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: baremetal.ironicproject.org/v1alpha1
2+
kind: IronicRunbook
3+
metadata:
4+
name: bmc-maintenance
5+
namespace: openstack
6+
spec:
7+
runbookName: bmc-maintenance
8+
description: "Performs BMC maintenance operations including clearing the job queue and synchronizing the BMC clock."
9+
traits:
10+
- CUSTOM_DELL_ABC
11+
- CUSTOM_DELL_XYZ
12+
13+
steps:
14+
- interface: management
15+
step: clear_job_queue
16+
order: 1
17+
18+
- interface: management
19+
step: set_bmc_clock
20+
order: 2

‎components/ironic/runbook-operator/role.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ metadata:
66
app.kubernetes.io/name: ironicrunbook
77
app.kubernetes.io/component: rbac
88
rules:
9-
# Read-only runbook permissions
9+
# Runbook permissions
1010
- apiGroups:
1111
- baremetal.ironicproject.org
1212
resources:
@@ -16,10 +16,12 @@ rules:
1616
- list
1717
- watch
1818

19-
# Read-only status permissions
19+
# Status update permissions
2020
- apiGroups:
2121
- baremetal.ironicproject.org
2222
resources:
2323
- ironicrunbooks/status
2424
verbs:
2525
- get
26+
- patch
27+
- update

‎components/ironic/runbook-operator/shell-operator-ironic.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ spec:
1717
restartPolicy: Always
1818
containers:
1919
- name: shell-operator
20-
image: ghcr.io/rackerlabs/understack/shell-operator-ironic:latest
20+
image: ghcr.io/rackerlabs/understack/shell-operator-ironic:pr-2051
2121
imagePullPolicy: Always
2222
env:
2323
- name: OS_CLOUD

‎containers/shell-operator-ironic/hooks/create_runbook.sh‎

Lines changed: 138 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -10,42 +10,158 @@ if [[ $1 == "--config" ]] ; then
1010
"executeHookOnEvent":[ "Added" ]
1111
}],
1212
"settings": {
13-
"executionMinInterval": 30s,
13+
"executionMinInterval": "30s",
1414
"executionBurst": 1
1515
}
1616
}
1717
EOF
1818
else
19+
patch_status() {
20+
local namespace="$1"
21+
local name="$2"
22+
local sync_status="$3"
23+
local message="${4:-}"
24+
local now
25+
now=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
26+
27+
local condition_status="True"
28+
local reason="SyncSucceeded"
29+
if [[ "${sync_status}" == "Failed" ]]; then
30+
condition_status="False"
31+
reason="SyncFailed"
32+
fi
33+
34+
local patch
35+
patch=$(cat <<PATCH
36+
{
37+
"status": {
38+
"syncStatus": "${sync_status}",
39+
"lastSyncTime": "${now}",
40+
"conditions": [
41+
{
42+
"type": "Ready",
43+
"status": "${condition_status}",
44+
"lastTransitionTime": "${now}",
45+
"reason": "${reason}",
46+
"message": "${message}"
47+
}
48+
]
49+
}
50+
}
51+
PATCH
52+
)
53+
echo "[create_runbook] Patching status namespace=${namespace} name=${name} syncStatus=${sync_status}"
54+
kubectl patch ironicrunbook "${name}" -n "${namespace}" \
55+
--type merge --subresource status -p "${patch}" 2>/dev/null || \
56+
echo "[create_runbook] WARNING: failed to patch status for ${name}"
57+
}
58+
59+
sync_runbook() {
60+
local i="$1"
61+
local resource_name namespace kind runbook_name description public owner
62+
63+
resource_name=$(jq -r ".${i}.metadata.name" "${BINDING_CONTEXT_PATH}")
64+
namespace=$(jq -r ".${i}.metadata.namespace" "${BINDING_CONTEXT_PATH}")
65+
kind=$(jq -r ".${i}.kind" "${BINDING_CONTEXT_PATH}")
66+
runbook_name=$(jq -r ".${i}.spec.runbookName" "${BINDING_CONTEXT_PATH}")
67+
description=$(jq -r ".${i}.spec.description // empty" "${BINDING_CONTEXT_PATH}")
68+
public=$(jq -r ".${i}.spec.public // empty" "${BINDING_CONTEXT_PATH}")
69+
owner=$(jq -r ".${i}.spec.owner // empty" "${BINDING_CONTEXT_PATH}")
70+
71+
echo "[create_runbook] Creating runbook kind=${kind} name=${resource_name} namespace=${namespace} runbookName=${runbook_name} description=${description} public=${public} owner=${owner}"
72+
73+
jq -r ".${i}.spec.steps" "${BINDING_CONTEXT_PATH}" > /tmp/steps.json
74+
75+
command_args=(baremetal runbook create --name "${runbook_name}" --steps /tmp/steps.json)
76+
77+
if [[ -n "${description}" ]]; then
78+
command_args+=(--description "${description}")
79+
fi
80+
if [[ -n "${public}" ]]; then
81+
command_args+=(--public "${public}")
82+
fi
83+
if [[ -n "${owner}" ]]; then
84+
command_args+=(--owner "${owner}")
85+
fi
86+
87+
echo "[create_runbook] Running: openstack ${command_args[*]}"
88+
89+
if output=$(openstack "${command_args[@]}" 2>&1); then
90+
echo "[create_runbook] SUCCESS: Runbook created in Ironic name=${resource_name} output=${output}"
91+
92+
traits_json=$(jq -c ".${i}.spec.traits // []" "${BINDING_CONTEXT_PATH}")
93+
if [[ "${traits_json}" != "[]" ]]; then
94+
echo "[create_runbook] Setting traits name=${resource_name} traits=${traits_json}"
95+
ironic_endpoint=$(openstack endpoint list --service baremetal --interface internal -f value -c URL 2>/dev/null | head -1)
96+
if [[ -n "${ironic_endpoint}" ]]; then
97+
token=$(openstack token issue -f value -c id)
98+
echo "[create_runbook] PUT ${ironic_endpoint}/v1/runbooks/${runbook_name}/traits"
99+
trait_response=$(curl -s -X PUT \
100+
-H "Content-Type: application/json" \
101+
-H "X-Auth-Token: ${token}" \
102+
-H "X-OpenStack-Ironic-API-Version: 1.112" \
103+
-d "{\"traits\": ${traits_json}}" \
104+
"${ironic_endpoint}/v1/runbooks/${runbook_name}/traits")
105+
echo "[create_runbook] Traits response name=${resource_name} response=${trait_response}"
106+
else
107+
echo "[create_runbook] WARNING: Could not determine Ironic endpoint for traits"
108+
fi
109+
else
110+
echo "[create_runbook] No traits to set name=${resource_name}"
111+
fi
112+
113+
patch_status "${namespace}" "${resource_name}" "Synced" "Successfully created runbook in Ironic"
114+
echo "[create_runbook] Completed name=${resource_name} status=Synced"
115+
else
116+
# If it already exists, that's OK during sync - not an error
117+
if echo "${output}" | grep -qi "already exists\|Conflict\|409"; then
118+
echo "[create_runbook] Runbook already exists in Ironic name=${resource_name}, skipping create"
119+
patch_status "${namespace}" "${resource_name}" "Synced" "Runbook already exists in Ironic"
120+
else
121+
echo "[create_runbook] FAILED: name=${resource_name} error=${output}" >&2
122+
patch_status "${namespace}" "${resource_name}" "Failed" "${output}"
123+
return 1
124+
fi
125+
fi
126+
}
127+
128+
echo "[create_runbook] Hook invoked, processing binding contexts"
19129
binding_count=$(jq -r 'length' "${BINDING_CONTEXT_PATH}")
130+
echo "[create_runbook] Found ${binding_count} binding context(s)"
131+
20132
for ((i = 0; i < binding_count; i++)); do
21133
type=$(jq -r ".[$i].type" "${BINDING_CONTEXT_PATH}")
134+
echo "[create_runbook] Processing context=${i} type=${type}"
135+
22136
if [[ $type == "Synchronization" ]] ; then
23-
echo "Implement any reconciliation logic needed here."
137+
echo "[create_runbook] Synchronization event, reconciling existing resources"
138+
objects_count=$(jq -r ".[$i].objects | length" "${BINDING_CONTEXT_PATH}")
139+
echo "[create_runbook] Found ${objects_count} existing IronicRunbook(s) to reconcile"
140+
for ((j = 0; j < objects_count; j++)); do
141+
obj_name=$(jq -r ".[$i].objects[$j].object.metadata.name" "${BINDING_CONTEXT_PATH}")
142+
obj_sync=$(jq -r ".[$i].objects[$j].object.status.syncStatus // empty" "${BINDING_CONTEXT_PATH}")
143+
echo "[create_runbook] Checking name=${obj_name} syncStatus=${obj_sync}"
144+
if [[ -z "${obj_sync}" || "${obj_sync}" == "null" ]]; then
145+
echo "[create_runbook] Resource name=${obj_name} has no syncStatus, needs reconciliation"
146+
# Re-map the jq path to point at the object within the sync event
147+
ORIG_BINDING_CONTEXT_PATH="${BINDING_CONTEXT_PATH}"
148+
jq -r ".[$i].objects[$j].object" "${BINDING_CONTEXT_PATH}" > /tmp/sync_object.json
149+
BINDING_CONTEXT_PATH=/tmp/sync_object.json
150+
sync_runbook ""
151+
BINDING_CONTEXT_PATH="${ORIG_BINDING_CONTEXT_PATH}"
152+
else
153+
echo "[create_runbook] Resource name=${obj_name} already synced, skipping"
154+
fi
155+
done
24156
continue
25157
fi
26158

27159
if [[ $type == "Event" ]] ; then
28-
resource_name=$(jq -r ".[$i].object.metadata.name" "${BINDING_CONTEXT_PATH}")
29-
kind=$(jq -r ".[$i].object.kind" "${BINDING_CONTEXT_PATH}")
30-
31-
runbook_name=$(jq -r ".[$i].object.spec.runbookName" "${BINDING_CONTEXT_PATH}")
32-
public=$(jq -r ".[$i].object.spec.public" "${BINDING_CONTEXT_PATH}")
33-
owner=$(jq -r ".[$i].object.spec.owner" "${BINDING_CONTEXT_PATH}")
34-
jq -r ".[$i].object.spec.steps" "${BINDING_CONTEXT_PATH}" > /tmp/steps.yaml
35-
36-
# Ironic's runbook extra field is essentially a dict of dicts, representing a key values. baremetal cli allows you
37-
# to pass in multiple --extra options, adding any you do pass. We would need to make an initial query to determine
38-
# existing extras, and then sync the differences. This work is probably better suited to a full controller implementation.
39-
# extra=$(jq -r '.spec.extra | [to_entries[] | "--extra \(.key)=\(.value | @json | @sh)"] | join(" ")' ${BINDING_CONTEXT_PATH})
40-
41-
command_args=(baremetal runbook create --name "${runbook_name}" --public "${public}" --steps /tmp/steps.yaml)
42-
if [[ -n "${owner}" && "${owner}" != "null" ]]; then
43-
command_args+=(--owner "${owner}")
160+
sync_runbook "[$i].object"
161+
if [[ $? -ne 0 ]]; then
162+
exit 1
44163
fi
45-
46-
echo "${kind}/${resource_name} created, running: openstack ${command_args[*]}"
47-
48-
openstack "${command_args[@]}"
49164
fi
50165
done
166+
echo "[create_runbook] Hook finished"
51167
fi

‎containers/shell-operator-ironic/hooks/delete_runbook.sh‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,30 +10,42 @@ if [[ $1 == "--config" ]] ; then
1010
"executeHookOnEvent":[ "Deleted" ]
1111
}],
1212
"settings": {
13-
"executionMinInterval": 30s,
13+
"executionMinInterval": "30s",
1414
"executionBurst": 1
1515
}
1616
}
1717
EOF
1818
else
19+
echo "[delete_runbook] Hook invoked, processing binding contexts"
1920
binding_count=$(jq -r 'length' "${BINDING_CONTEXT_PATH}")
21+
echo "[delete_runbook] Found ${binding_count} binding context(s)"
22+
2023
for ((i = 0; i < binding_count; i++)); do
2124
type=$(jq -r ".[$i].type" "${BINDING_CONTEXT_PATH}")
25+
echo "[delete_runbook] Processing context=${i} type=${type}"
26+
2227
if [[ $type == "Synchronization" ]] ; then
23-
echo "Implement any reconciliation logic needed here."
28+
echo "[delete_runbook] Synchronization event, nothing to do for deletes"
2429
continue
2530
fi
2631

2732
if [[ $type == "Event" ]] ; then
2833
resource_name=$(jq -r ".[$i].object.metadata.name" "${BINDING_CONTEXT_PATH}")
2934
kind=$(jq -r ".[$i].object.kind" "${BINDING_CONTEXT_PATH}")
30-
3135
runbook_name=$(jq -r ".[$i].object.spec.runbookName" "${BINDING_CONTEXT_PATH}")
3236

37+
echo "[delete_runbook] Deleting runbook kind=${kind} name=${resource_name} runbookName=${runbook_name}"
38+
3339
command_args=(baremetal runbook delete "${runbook_name}")
34-
echo "${kind}/${resource_name} deleted, running: openstack ${command_args[*]}"
40+
echo "[delete_runbook] Running: openstack ${command_args[*]}"
3541

36-
openstack "${command_args[@]}"
42+
if output=$(openstack "${command_args[@]}" 2>&1); then
43+
echo "[delete_runbook] SUCCESS: Runbook deleted from Ironic name=${resource_name} output=${output}"
44+
else
45+
echo "[delete_runbook] FAILED: name=${resource_name} error=${output}" >&2
46+
exit 1
47+
fi
3748
fi
3849
done
50+
echo "[delete_runbook] Hook finished"
3951
fi

0 commit comments

Comments
 (0)