Skip to content

Commit 4869a26

Browse files
shayan1995shayan1995
authored
fix(helm): correct boot-artifacts mount path and remove readOnly (#2813)
<!-- Describe what this PR does --> The helm charts mounted the boot-artifacts volume at /boot-artifacts/blobs/internal with readOnly: true on the main container. The Rust code hardcodes /forge-boot-artifacts/blobs/internal (matching the forged kustomize overlays which work in production), and two code paths write into the directory at runtime. This caused DPU BFB recovery to fail on every helm-deployed site. ## Related issues #2490 ## Type of Change - [ ] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [x] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [x] Unit tests added/updated - [ ] Integration tests added/updated - [ ] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes Two bugs fixed, both caused by the helm chart diverging from the forged kustomize overlays (the production reference implementation): **1. Wrong mount path (`/boot-artifacts` → `/forge-boot-artifacts`)** The Rust code hardcodes `/forge-boot-artifacts/blobs/internal` in three places: - `crates/preingestion-manager/src/bfb_rshim_copier.rs` — `PREINGESTION_BFB_PATH` and `UNIFIED_PREINGESTION_BFB_PATH` - `crates/machine-controller/src/handler.rs` — secure boot certificate path The forged kustomize overlay (`components/boot-artifacts-containers/patch-boot-artifacts.yaml`) uses the same path and works correctly in production. The helm chart used `/boot-artifacts/blobs/internal`, causing a "No such file or directory" error on every BFB copy attempt in helm-deployed sites. **2. `readOnly: true` on a directory the code writes into** The main container mount had `readOnly: true`. Two runtime code paths write into this directory: - BFB recovery writes `preingestion_unified_update.bfb` before streaming it to the DPU BMC rshim - Rack firmware update uses the directory as a download cache All 35 helm unit tests pass.
1 parent 874659f commit 4869a26

8 files changed

Lines changed: 50 additions & 56 deletions

File tree

‎deploy/components/boot-artifacts-containers/patch-boot-artifacts.yaml‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
path: /spec/template/spec/containers/0/volumeMounts/0
2424
value:
2525
name: boot-artifacts
26-
mountPath: /nico-boot-artifacts/blobs/internal
26+
mountPath: /forge-boot-artifacts/blobs/internal
2727

2828
- op: add
2929
path: /spec/template/spec/containers/0
@@ -32,21 +32,21 @@
3232
image: yourdockerregistry.com/path/to/boot-artifacts-x86_64:latest
3333
imagePullPolicy: IfNotPresent
3434
command: ["/bin/sh"]
35-
args: ["-c", "cp -r /x86_64 /nico-boot-artifacts/blobs/internal; trap : TERM INT; sleep 9999999999d & wait"]
35+
args: ["-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal; trap : TERM INT; sleep 9999999999d & wait"]
3636
volumeMounts:
3737
- name: boot-artifacts
38-
mountPath: /nico-boot-artifacts/blobs/internal
38+
mountPath: /forge-boot-artifacts/blobs/internal
3939
- op: add
4040
path: /spec/template/spec/containers/0
4141
value:
4242
name: boot-artifacts-aarch64
4343
image: yourdockerregistry.com/path/to/boot-artifacts-aarch64:latest
4444
imagePullPolicy: IfNotPresent
4545
command: ["/bin/sh"]
46-
args: ["-c", "cp -r /aarch64 /nico-boot-artifacts/blobs/internal; cp -r /apt /nico-boot-artifacts/blobs/internal; cp -r /firmware /nico-boot-artifacts/blobs/internal; trap : TERM INT; sleep 9999999999d & wait"]
46+
args: ["-c", "cp -r /aarch64 /forge-boot-artifacts/blobs/internal; cp -r /apt /forge-boot-artifacts/blobs/internal; cp -r /firmware /forge-boot-artifacts/blobs/internal; trap : TERM INT; sleep 9999999999d & wait"]
4747
volumeMounts:
4848
- name: boot-artifacts
49-
mountPath: /nico-boot-artifacts/blobs/internal
49+
mountPath: /forge-boot-artifacts/blobs/internal
5050

5151
# We keep this image around to still provide access to the previous
5252
# qcow-imager.efi/root binaries for backward compatibility, this should be
@@ -58,18 +58,18 @@
5858
image: yourdockerregistry.com/path/to/boot-artifacts-x86_64:v2023.11-rc1-16-gf7aebdcc
5959
imagePullPolicy: IfNotPresent
6060
command: ["/bin/sh"]
61-
args: ["-c", "mkdir -vp /nico-boot-artifacts/blobs/internal/x86_64 && cp -fr /x86_64/qcow-imaging* /nico-boot-artifacts/blobs/internal/x86_64; trap : TERM INT; sleep 9999999999d & wait"]
61+
args: ["-c", "mkdir -vp /forge-boot-artifacts/blobs/internal/x86_64 && cp -fr /x86_64/qcow-imaging* /forge-boot-artifacts/blobs/internal/x86_64; trap : TERM INT; sleep 9999999999d & wait"]
6262
volumeMounts:
6363
- name: boot-artifacts
64-
mountPath: /nico-boot-artifacts/blobs/internal
64+
mountPath: /forge-boot-artifacts/blobs/internal
6565
- op: add
6666
path: /spec/template/spec/containers/0
6767
value:
6868
name: machine-validation-artifacts-config
6969
image: yourdockerregistry.com/path/to/nvmetal-scout-burn-in:1.3.0
7070
imagePullPolicy: IfNotPresent
7171
command: ["/bin/sh"]
72-
args: ["-c", "cp -r /machine-validation /nico-boot-artifacts/blobs/internal; trap : TERM INT; sleep 9999999999d & wait"]
72+
args: ["-c", "cp -r /machine-validation /forge-boot-artifacts/blobs/internal; trap : TERM INT; sleep 9999999999d & wait"]
7373
volumeMounts:
7474
- name: boot-artifacts
75-
mountPath: /nico-boot-artifacts/blobs/internal
75+
mountPath: /forge-boot-artifacts/blobs/internal

‎helm/charts/nico-api/templates/deployment.yaml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ spec:
4141
{{- end }}
4242
volumeMounts:
4343
- name: boot-artifacts
44-
mountPath: /boot-artifacts/blobs/internal
44+
mountPath: /forge-boot-artifacts/blobs/internal
4545
{{- with .volumeMounts }}
4646
{{- toYaml . | nindent 12 }}
4747
{{- end }}
@@ -187,8 +187,7 @@ spec:
187187
readOnly: true
188188
{{- if .Values.bootArtifactContainers }}
189189
- name: boot-artifacts
190-
mountPath: /boot-artifacts/blobs/internal
191-
readOnly: true
190+
mountPath: /forge-boot-artifacts/blobs/internal
192191
{{- end }}
193192
resources:
194193
{{- toYaml .Values.resources | nindent 12 }}

‎helm/charts/nico-api/tests/boot_artifacts_test.yaml‎

Lines changed: 10 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,17 @@ tests:
1717
path: spec.template.spec.containers[0].volumeMounts
1818
content:
1919
name: boot-artifacts
20-
mountPath: /boot-artifacts/blobs/internal
21-
readOnly: true
20+
mountPath: /forge-boot-artifacts/blobs/internal
2221

2322
- it: should render boot artifact init containers with auto-injected volume mount
2423
set:
2524
bootArtifactContainers:
2625
- name: boot-artifacts-x86-64
2726
image: registry.example.com/boot-artifacts-x86_64:latest
28-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
27+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
2928
- name: boot-artifacts-aarch64
3029
image: registry.example.com/boot-artifacts-aarch64:latest
31-
command: ["sh", "-c", "cp -r /aarch64 /apt /firmware /boot-artifacts/blobs/internal"]
30+
command: ["sh", "-c", "cp -r /aarch64 /apt /firmware /forge-boot-artifacts/blobs/internal"]
3231
asserts:
3332
- exists:
3433
path: spec.template.spec.initContainers
@@ -45,19 +44,19 @@ tests:
4544
path: spec.template.spec.initContainers[0].volumeMounts
4645
content:
4746
name: boot-artifacts
48-
mountPath: /boot-artifacts/blobs/internal
47+
mountPath: /forge-boot-artifacts/blobs/internal
4948
- contains:
5049
path: spec.template.spec.initContainers[1].volumeMounts
5150
content:
5251
name: boot-artifacts
53-
mountPath: /boot-artifacts/blobs/internal
52+
mountPath: /forge-boot-artifacts/blobs/internal
5453

5554
- it: should add boot-artifacts volume and mount on main container
5655
set:
5756
bootArtifactContainers:
5857
- name: boot-artifacts-x86-64
5958
image: registry.example.com/boot-artifacts-x86_64:latest
60-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
59+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
6160
asserts:
6261
- contains:
6362
path: spec.template.spec.volumes
@@ -68,15 +67,14 @@ tests:
6867
path: spec.template.spec.containers[0].volumeMounts
6968
content:
7069
name: boot-artifacts
71-
mountPath: /boot-artifacts/blobs/internal
72-
readOnly: true
70+
mountPath: /forge-boot-artifacts/blobs/internal
7371

7472
- it: should merge bootArtifactContainers and initContainers
7573
set:
7674
bootArtifactContainers:
7775
- name: boot-artifacts-x86-64
7876
image: registry.example.com/boot-artifacts-x86_64:latest
79-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
77+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
8078
initContainers:
8179
- name: custom-init
8280
image: busybox:latest
@@ -97,7 +95,7 @@ tests:
9795
bootArtifactContainers:
9896
- name: boot-artifacts-x86-64
9997
image: registry.example.com/boot-artifacts-x86_64:latest
100-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
98+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
10199
volumeMounts:
102100
- name: extra-volume
103101
mountPath: /extra
@@ -109,7 +107,7 @@ tests:
109107
path: spec.template.spec.initContainers[0].volumeMounts
110108
content:
111109
name: boot-artifacts
112-
mountPath: /boot-artifacts/blobs/internal
110+
mountPath: /forge-boot-artifacts/blobs/internal
113111
- contains:
114112
path: spec.template.spec.initContainers[0].volumeMounts
115113
content:

‎helm/charts/nico-api/values.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,19 +79,19 @@ initContainers: []
7979

8080
## Boot-artifact init containers
8181
## Each entry is rendered as a Kubernetes init container with the boot-artifacts
82-
## volume automatically mounted at /boot-artifacts/blobs/internal.
82+
## volume automatically mounted at /forge-boot-artifacts/blobs/internal.
8383
bootArtifactContainers: []
8484
# bootArtifactContainers:
8585
# - name: boot-artifacts-x86-64
8686
# image: <your-registry>/boot-artifacts-x86_64:latest
87-
# command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
87+
# command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
8888
# - name: boot-artifacts-aarch64
8989
# image: <your-registry>/boot-artifacts-aarch64:latest
9090
# # NOTE: the aarch64 image ships /aarch64 + /apt (NOT /firmware) — copying a missing dir crash-loops the init.
91-
# command: ["sh", "-c", "cp -r /aarch64 /apt /boot-artifacts/blobs/internal"]
91+
# command: ["sh", "-c", "cp -r /aarch64 /apt /forge-boot-artifacts/blobs/internal"]
9292
# - name: machine-validation-config
9393
# image: <your-registry>/machine-validation-config:latest
94-
# command: ["sh", "-c", "cp -r /machine-validation /boot-artifacts/blobs/internal"]
94+
# command: ["sh", "-c", "cp -r /machine-validation /forge-boot-artifacts/blobs/internal"]
9595

9696
service:
9797
grpc:

‎helm/charts/nico-pxe/templates/deployment.yaml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ spec:
4040
{{- end }}
4141
volumeMounts:
4242
- name: boot-artifacts
43-
mountPath: /boot-artifacts/blobs/internal
43+
mountPath: /forge-boot-artifacts/blobs/internal
4444
{{- with .volumeMounts }}
4545
{{- toYaml . | nindent 12 }}
4646
{{- end }}
@@ -86,8 +86,7 @@ spec:
8686
readOnly: true
8787
{{- if .Values.bootArtifactContainers }}
8888
- name: boot-artifacts
89-
mountPath: /boot-artifacts/blobs/internal
90-
readOnly: true
89+
mountPath: /forge-boot-artifacts/blobs/internal
9190
{{- end }}
9291
volumes:
9392
- name: spiffe

‎helm/charts/nico-pxe/tests/boot_artifacts_test.yaml‎

Lines changed: 10 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,17 @@ tests:
1717
path: spec.template.spec.containers[0].volumeMounts
1818
content:
1919
name: boot-artifacts
20-
mountPath: /boot-artifacts/blobs/internal
21-
readOnly: true
20+
mountPath: /forge-boot-artifacts/blobs/internal
2221

2322
- it: should render boot artifact init containers with auto-injected volume mount
2423
set:
2524
bootArtifactContainers:
2625
- name: boot-artifacts-x86-64
2726
image: registry.example.com/boot-artifacts-x86_64:latest
28-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
27+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
2928
- name: boot-artifacts-aarch64
3029
image: registry.example.com/boot-artifacts-aarch64:latest
31-
command: ["sh", "-c", "cp -r /aarch64 /apt /firmware /boot-artifacts/blobs/internal"]
30+
command: ["sh", "-c", "cp -r /aarch64 /apt /firmware /forge-boot-artifacts/blobs/internal"]
3231
asserts:
3332
- exists:
3433
path: spec.template.spec.initContainers
@@ -42,19 +41,19 @@ tests:
4241
path: spec.template.spec.initContainers[0].volumeMounts
4342
content:
4443
name: boot-artifacts
45-
mountPath: /boot-artifacts/blobs/internal
44+
mountPath: /forge-boot-artifacts/blobs/internal
4645
- contains:
4746
path: spec.template.spec.initContainers[1].volumeMounts
4847
content:
4948
name: boot-artifacts
50-
mountPath: /boot-artifacts/blobs/internal
49+
mountPath: /forge-boot-artifacts/blobs/internal
5150

5251
- it: should add boot-artifacts volume and mount on main container
5352
set:
5453
bootArtifactContainers:
5554
- name: boot-artifacts-x86-64
5655
image: registry.example.com/boot-artifacts-x86_64:latest
57-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
56+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
5857
asserts:
5958
- contains:
6059
path: spec.template.spec.volumes
@@ -65,8 +64,7 @@ tests:
6564
path: spec.template.spec.containers[0].volumeMounts
6665
content:
6766
name: boot-artifacts
68-
mountPath: /boot-artifacts/blobs/internal
69-
readOnly: true
67+
mountPath: /forge-boot-artifacts/blobs/internal
7068

7169
- it: should use configurable serve path
7270
set:
@@ -79,18 +77,18 @@ tests:
7977
path: spec.template.spec.containers[0].command[2]
8078
pattern: '-s /custom-boot-path'
8179

82-
- it: should default serve path to /boot-artifacts
80+
- it: should default serve path to /forge-boot-artifacts
8381
asserts:
8482
- matchRegex:
8583
path: spec.template.spec.containers[0].command[2]
86-
pattern: '-s /boot-artifacts'
84+
pattern: '-s /forge-boot-artifacts'
8785

8886
- it: should merge bootArtifactContainers and initContainers
8987
set:
9088
bootArtifactContainers:
9189
- name: boot-artifacts-x86-64
9290
image: registry.example.com/boot-artifacts-x86_64:latest
93-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
91+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
9492
initContainers:
9593
- name: custom-init
9694
image: busybox:latest

‎helm/charts/nico-pxe/values.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ initContainers: []
5050

5151
## Boot-artifact init containers
5252
## Each entry is rendered as a Kubernetes init container with the boot-artifacts
53-
## volume automatically mounted at /boot-artifacts/blobs/internal, then served by
53+
## volume automatically mounted at /forge-boot-artifacts/blobs/internal, then served by
5454
## nico-pxe at /public/blobs/internal.
5555
##
5656
## REQUIRED for DPU and host HTTP boot: if empty, the DPU/host requests its bootloader
@@ -64,17 +64,17 @@ bootArtifactContainers: []
6464
# bootArtifactContainers:
6565
# - name: boot-artifacts-x86-64
6666
# image: <your-registry>/boot-artifacts-x86_64:<tag>
67-
# command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
67+
# command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
6868
# - name: boot-artifacts-aarch64
6969
# image: <your-registry>/boot-artifacts-aarch64:<tag>
70-
# command: ["sh", "-c", "cp -r /aarch64 /apt /boot-artifacts/blobs/internal"]
70+
# command: ["sh", "-c", "cp -r /aarch64 /apt /forge-boot-artifacts/blobs/internal"]
7171
# - name: machine-validation-config
7272
# image: <your-registry>/machine-validation-config:<tag>
73-
# command: ["sh", "-c", "cp -r /machine-validation /boot-artifacts/blobs/internal"]
73+
# command: ["sh", "-c", "cp -r /machine-validation /forge-boot-artifacts/blobs/internal"]
7474

7575
## Boot artifact serving configuration
7676
bootArtifacts:
77-
servePath: /boot-artifacts
77+
servePath: /forge-boot-artifacts
7878

7979
annotations:
8080
configmap.reloader.stakater.com/reload: '{{ include "nico-pxe.name" . }}-env-config'

‎helm/examples/values-full.yaml‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -55,18 +55,18 @@ nico-api:
5555
cpu: 1500m
5656
memory: 8Gi
5757

58-
## Boot-artifact init containers — volume mount at /boot-artifacts/blobs/internal
58+
## Boot-artifact init containers — volume mount at /forge-boot-artifacts/blobs/internal
5959
## is auto-injected by the template.
6060
bootArtifactContainers:
6161
- name: boot-artifacts-x86-64
6262
image: "your-registry.example.com/boot-artifacts-x86_64:latest"
63-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
63+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
6464
- name: boot-artifacts-aarch64
6565
image: "your-registry.example.com/boot-artifacts-aarch64:latest"
66-
command: ["sh", "-c", "cp -r /aarch64 /apt /firmware /boot-artifacts/blobs/internal"]
66+
command: ["sh", "-c", "cp -r /aarch64 /apt /forge-boot-artifacts/blobs/internal"]
6767
- name: machine-validation-config
6868
image: "your-registry.example.com/machine-validation-config:latest"
69-
command: ["sh", "-c", "cp -r /machine-validation /boot-artifacts/blobs/internal"]
69+
command: ["sh", "-c", "cp -r /machine-validation /forge-boot-artifacts/blobs/internal"]
7070

7171
env:
7272
VAULT_PKI_ROLE_NAME: "nico-cluster"
@@ -240,18 +240,18 @@ nico-pxe:
240240
replicas: 1
241241

242242
bootArtifacts:
243-
servePath: /boot-artifacts
243+
servePath: /forge-boot-artifacts
244244

245245
bootArtifactContainers:
246246
- name: boot-artifacts-x86-64
247247
image: "your-registry.example.com/boot-artifacts-x86_64:latest"
248-
command: ["sh", "-c", "cp -r /x86_64 /boot-artifacts/blobs/internal"]
248+
command: ["sh", "-c", "cp -r /x86_64 /forge-boot-artifacts/blobs/internal"]
249249
- name: boot-artifacts-aarch64
250250
image: "your-registry.example.com/boot-artifacts-aarch64:latest"
251-
command: ["sh", "-c", "cp -r /aarch64 /apt /firmware /boot-artifacts/blobs/internal"]
251+
command: ["sh", "-c", "cp -r /aarch64 /apt /forge-boot-artifacts/blobs/internal"]
252252
- name: machine-validation-config
253253
image: "your-registry.example.com/machine-validation-config:latest"
254-
command: ["sh", "-c", "cp -r /machine-validation /boot-artifacts/blobs/internal"]
254+
command: ["sh", "-c", "cp -r /machine-validation /forge-boot-artifacts/blobs/internal"]
255255

256256
externalService:
257257
enabled: true

0 commit comments

Comments
 (0)