Skip to content

Commit 2cd0629

Browse files
roderikgemini-code-assist[bot]sourcery-ai[bot]
authored
docs: add pvc resize runbook (#28)
## Summary - add StatefulSet PVC scaling runbook to README/README.tpl - document helm values update and kubectl patch loop for resizing ## Testing - bun run check - bun run typecheck - bun test ## Summary by Sourcery Add a runbook section to the README and its template that guides users through scaling StatefulSet PVC storage without recreating pods. Documentation: - Add a new "Scale StatefulSet PVC storage (runbook)" section to README and README.tpl. - Document steps to update Helm values, roll out changes with helm upgrade, patch PVCs using kubectl, and verify the resize. - Include a note on enabling allowVolumeExpansion in the StorageClass if needed. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
1 parent d06bdbf commit 2cd0629

3 files changed

Lines changed: 131 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,66 @@ rm values-external.yaml
112112

113113
Summon resolves the secrets in memory, `envsubst` renders them into a transient values file, and Helm creates the ConfigMaps/Secrets required by the Besu nodes. The temporary file is removed once the release is installed.
114114

115+
### Scale StatefulSet PVC storage (runbook)
116+
117+
Use this runbook to grow the validator and RPC data volumes without recreating the StatefulSets.
118+
119+
1. Edit your Helm values so new pods request the larger capacity and keep the updated defaults:
120+
121+
```yaml
122+
network-nodes:
123+
persistence:
124+
enabled: true
125+
storageClass: fast-ssd # cluster storage class that supports expansion
126+
size: 200Gi # target size for every validator/RPC PVC
127+
retention:
128+
whenDeleted: Retain
129+
whenScaled: Retain
130+
```
131+
132+
2. Roll the values into the release (reuse your existing overrides):
133+
134+
```bash
135+
RELEASE="besu-network"
136+
NAMESPACE="besu"
137+
138+
helm upgrade --install "${RELEASE}" ./charts/network \
139+
--namespace "${NAMESPACE}" \
140+
--values values.yaml
141+
```
142+
143+
3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows. The loop echoes success or failure for each PVC—investigate any errors (insufficient quota, permissions, driver limits) before proceeding:
144+
145+
```bash
146+
# IMPORTANT: Set this to the same value as `network-nodes.persistence.size` from step 1.
147+
NEW_SIZE="200Gi"
148+
RELEASE="besu-network"
149+
NAMESPACE="besu"
150+
151+
for component in validator rpc; do
152+
kubectl get pvc -n "${NAMESPACE}" \
153+
-l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \
154+
-o name \
155+
| while read -r pvc; do
156+
if kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \
157+
-p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"; then
158+
echo "Successfully patched ${pvc}"
159+
else
160+
echo "ERROR: Failed to patch ${pvc}" >&2
161+
fi
162+
done
163+
done
164+
```
165+
166+
4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize):
167+
168+
> **Note:** The `FileSystemResizePending` status typically clears within a few minutes, but may take up to 10–15 minutes depending on your storage backend and cluster load. If the status persists longer than expected, check your CSI driver logs and node status for issues. For troubleshooting, see [Kubernetes PVC resizing documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims).
169+
170+
```bash
171+
kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w
172+
173+
If the StorageClass sets `allowVolumeExpansion: false`, patch it to `true` before running the loop or redeploy with a class that supports online resizing.
174+
115175
### Local artefact generation with Docker
116176

117177
Run the bootstrapper container locally to capture all artefacts before loading them into Conjur or another secret manager.

‎README.tpl‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,66 @@ rm values-external.yaml
112112

113113
Summon resolves the secrets in memory, `envsubst` renders them into a transient values file, and Helm creates the ConfigMaps/Secrets required by the Besu nodes. The temporary file is removed once the release is installed.
114114

115+
### Scale StatefulSet PVC storage (runbook)
116+
117+
Use this runbook to grow the validator and RPC data volumes without recreating the StatefulSets.
118+
119+
1. Edit your Helm values so new pods request the larger capacity and keep the updated defaults:
120+
121+
```yaml
122+
network-nodes:
123+
persistence:
124+
enabled: true
125+
storageClass: fast-ssd # cluster storage class that supports expansion
126+
size: 200Gi # target size for every validator/RPC PVC
127+
retention:
128+
whenDeleted: Retain
129+
whenScaled: Retain
130+
```
131+
132+
2. Roll the values into the release (reuse your existing overrides):
133+
134+
```bash
135+
RELEASE="besu-network"
136+
NAMESPACE="besu"
137+
138+
helm upgrade --install "${RELEASE}" ./charts/network \
139+
--namespace "${NAMESPACE}" \
140+
--values values.yaml
141+
```
142+
143+
3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows. The loop echoes success or failure for each PVC—investigate any errors (insufficient quota, permissions, driver limits) before proceeding:
144+
145+
```bash
146+
# IMPORTANT: Set this to the same value as `network-nodes.persistence.size` from step 1.
147+
NEW_SIZE="200Gi"
148+
RELEASE="besu-network"
149+
NAMESPACE="besu"
150+
151+
for component in validator rpc; do
152+
kubectl get pvc -n "${NAMESPACE}" \
153+
-l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \
154+
-o name \
155+
| while read -r pvc; do
156+
if kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \
157+
-p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"; then
158+
echo "Successfully patched ${pvc}"
159+
else
160+
echo "ERROR: Failed to patch ${pvc}" >&2
161+
fi
162+
done
163+
done
164+
```
165+
166+
4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize):
167+
168+
> **Note:** The `FileSystemResizePending` status typically clears within a few minutes, but may take up to 10–15 minutes depending on your storage backend and cluster load. If the status persists longer than expected, check your CSI driver logs and node status for issues. For troubleshooting, see [Kubernetes PVC resizing documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims).
169+
170+
```bash
171+
kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w
172+
173+
If the StorageClass sets `allowVolumeExpansion: false`, patch it to `true` before running the loop or redeploy with a class that supports online resizing.
174+
115175
### Local artefact generation with Docker
116176
117177
Run the bootstrapper container locally to capture all artefacts before loading them into Conjur or another secret manager.

‎lefthook.yml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,17 @@ pre-commit:
44
run: bun run check:fix || true
55
stage_fixed: true
66
docs_cli:
7-
run: bun run docs:cli || true
7+
run: |
8+
if bun run docs:cli; then
9+
git add README.md
10+
fi
811
stage_fixed: true
912
docs_helm:
10-
run: bun run docs:helm || true
13+
run: |
14+
if bun run docs:helm; then
15+
git add \
16+
charts/network/README.md \
17+
charts/network/charts/network-bootstrapper/README.md \
18+
charts/network/charts/network-nodes/README.md
19+
fi
1120
stage_fixed: true

0 commit comments

Comments
 (0)