From 95f31325af87acc60c2421fe581399e076c512ab Mon Sep 17 00:00:00 2001 From: Roderik van der Veer Date: Thu, 18 Sep 2025 08:50:21 +0200 Subject: [PATCH 1/6] docs(readme): add pvc resize runbook --- README.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ README.tpl | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+) diff --git a/README.md b/README.md index ba3df41..55644f2 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,60 @@ rm values-external.yaml Summon resolves the secrets in memory, `envsubst` renders them into a transient values file, and Helm creates the ConfigMaps/Secrets required by the Besu nodes. The temporary file is removed once the release is installed. +### Scale StatefulSet PVC storage (runbook) + +Use this runbook to grow the validator and RPC data volumes without recreating the StatefulSets. + +1. Edit your Helm values so new pods request the larger capacity and keep the updated defaults: + + ```yaml + network-nodes: + persistence: + enabled: true + storageClass: fast-ssd # cluster storage class that supports expansion + size: 200Gi # target size for every validator/RPC PVC + retention: + whenDeleted: Retain + whenScaled: Retain + ``` + +2. Roll the values into the release (reuse your existing overrides): + + ```bash + RELEASE="besu-network" + NAMESPACE="besu" + + helm upgrade --install "${RELEASE}" ./charts/network \ + --namespace "${NAMESPACE}" \ + --values values.yaml + ``` + +3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: + + ```bash + NEW_SIZE="200Gi" + RELEASE="besu-network" + NAMESPACE="besu" + + for component in validator rpc; do + kubectl get pvc -n "${NAMESPACE}" \ + -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \ + -o name \ + | while read -r pvc; do + kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \ + -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}" + done + done + ``` + +4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize): + + ```bash + kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w + ``` + +If the StorageClass sets `allowVolumeExpansion: false`, patch it to `true` before running the loop or redeploy with a class that supports online resizing. + ### Local artefact generation with Docker Run the bootstrapper container locally to capture all artefacts before loading them into Conjur or another secret manager. diff --git a/README.tpl b/README.tpl index 040b01c..dde1e98 100644 --- a/README.tpl +++ b/README.tpl @@ -112,6 +112,60 @@ rm values-external.yaml Summon resolves the secrets in memory, `envsubst` renders them into a transient values file, and Helm creates the ConfigMaps/Secrets required by the Besu nodes. The temporary file is removed once the release is installed. +### Scale StatefulSet PVC storage (runbook) + +Use this runbook to grow the validator and RPC data volumes without recreating the StatefulSets. + +1. Edit your Helm values so new pods request the larger capacity and keep the updated defaults: + + ```yaml + network-nodes: + persistence: + enabled: true + storageClass: fast-ssd # cluster storage class that supports expansion + size: 200Gi # target size for every validator/RPC PVC + retention: + whenDeleted: Retain + whenScaled: Retain + ``` + +2. Roll the values into the release (reuse your existing overrides): + + ```bash + RELEASE="besu-network" + NAMESPACE="besu" + + helm upgrade --install "${RELEASE}" ./charts/network \ + --namespace "${NAMESPACE}" \ + --values values.yaml + ``` + +3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: + + ```bash + NEW_SIZE="200Gi" + RELEASE="besu-network" + NAMESPACE="besu" + + for component in validator rpc; do + kubectl get pvc -n "${NAMESPACE}" \ + -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \ + -o name \ + | while read -r pvc; do + kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \ + -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}" + done + done + ``` + +4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize): + + ```bash + kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w + ``` + +If the StorageClass sets `allowVolumeExpansion: false`, patch it to `true` before running the loop or redeploy with a class that supports online resizing. + ### Local artefact generation with Docker Run the bootstrapper container locally to capture all artefacts before loading them into Conjur or another secret manager. From d061756ab7746680e8976a601eff64ce61ce2600 Mon Sep 17 00:00:00 2001 From: Roderik van der Veer Date: Thu, 18 Sep 2025 08:53:10 +0200 Subject: [PATCH 2/6] Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 55644f2..b48a7e3 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,7 @@ Use this runbook to grow the validator and RPC data volumes without recreating t 3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: ```bash + # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` in your Helm values from step 1. NEW_SIZE="200Gi" RELEASE="besu-network" NAMESPACE="besu" From de6766d0007d2038b07e4edfb207034ab135bd40 Mon Sep 17 00:00:00 2001 From: Roderik van der Veer Date: Thu, 18 Sep 2025 08:53:28 +0200 Subject: [PATCH 3/6] Apply suggestion from @sourcery-ai[bot] Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- README.tpl | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.tpl b/README.tpl index dde1e98..fbbe5ea 100644 --- a/README.tpl +++ b/README.tpl @@ -160,9 +160,10 @@ Use this runbook to grow the validator and RPC data volumes without recreating t 4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize): + > **Note:** The `FileSystemResizePending` status typically clears within a few minutes, but may take up to 10–15 minutes depending on your storage backend and cluster load. If the status persists longer than expected, check your CSI driver logs and node status for issues. For troubleshooting, see [Kubernetes PVC resizing documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims). + ```bash kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w - ``` If the StorageClass sets `allowVolumeExpansion: false`, patch it to `true` before running the loop or redeploy with a class that supports online resizing. From f4486ebc551a8491097145634bd87cd348c76133 Mon Sep 17 00:00:00 2001 From: Roderik van der Veer Date: Thu, 18 Sep 2025 08:54:11 +0200 Subject: [PATCH 4/6] chore(lefthook): auto-stage generated docs --- lefthook.yml | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/lefthook.yml b/lefthook.yml index 793d802..8c9070b 100644 --- a/lefthook.yml +++ b/lefthook.yml @@ -4,8 +4,17 @@ pre-commit: run: bun run check:fix || true stage_fixed: true docs_cli: - run: bun run docs:cli || true + run: | + if bun run docs:cli; then + git add README.md + fi stage_fixed: true docs_helm: - run: bun run docs:helm || true + run: | + if bun run docs:helm; then + git add \ + charts/network/README.md \ + charts/network/charts/network-bootstrapper/README.md \ + charts/network/charts/network-nodes/README.md + fi stage_fixed: true From 321f58d842c66b783ba5b46802c195a8f95cec81 Mon Sep 17 00:00:00 2001 From: Roderik van der Veer Date: Thu, 18 Sep 2025 08:56:12 +0200 Subject: [PATCH 5/6] docs(readme): remind pvc size sync --- README.md | 5 +++-- README.tpl | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b48a7e3..276b570 100644 --- a/README.md +++ b/README.md @@ -143,7 +143,7 @@ Use this runbook to grow the validator and RPC data volumes without recreating t 3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: ```bash - # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` in your Helm values from step 1. + # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` from step 1. NEW_SIZE="200Gi" RELEASE="besu-network" NAMESPACE="besu" @@ -161,9 +161,10 @@ Use this runbook to grow the validator and RPC data volumes without recreating t 4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize): + > **Note:** The `FileSystemResizePending` status typically clears within a few minutes, but may take up to 10–15 minutes depending on your storage backend and cluster load. If the status persists longer than expected, check your CSI driver logs and node status for issues. For troubleshooting, see [Kubernetes PVC resizing documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims). + ```bash kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w - ``` If the StorageClass sets `allowVolumeExpansion: false`, patch it to `true` before running the loop or redeploy with a class that supports online resizing. diff --git a/README.tpl b/README.tpl index fbbe5ea..7eb0e81 100644 --- a/README.tpl +++ b/README.tpl @@ -143,6 +143,7 @@ Use this runbook to grow the validator and RPC data volumes without recreating t 3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: ```bash + # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` from step 1. NEW_SIZE="200Gi" RELEASE="besu-network" NAMESPACE="besu" From 0abadd1b7c1ab28d37012c492a91ee95a0285a8c Mon Sep 17 00:00:00 2001 From: Roderik van der Veer Date: Thu, 18 Sep 2025 08:57:37 +0200 Subject: [PATCH 6/6] docs(readme): log pvc patch failures --- README.md | 10 +++++++--- README.tpl | 10 +++++++--- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 276b570..fe21c8c 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ Use this runbook to grow the validator and RPC data volumes without recreating t --values values.yaml ``` -3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: +3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows. The loop echoes success or failure for each PVC—investigate any errors (insufficient quota, permissions, driver limits) before proceeding: ```bash # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` from step 1. @@ -153,8 +153,12 @@ Use this runbook to grow the validator and RPC data volumes without recreating t -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \ -o name \ | while read -r pvc; do - kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \ - -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}" + if kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \ + -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"; then + echo "Successfully patched ${pvc}" + else + echo "ERROR: Failed to patch ${pvc}" >&2 + fi done done ``` diff --git a/README.tpl b/README.tpl index 7eb0e81..fa2e1f5 100644 --- a/README.tpl +++ b/README.tpl @@ -140,7 +140,7 @@ Use this runbook to grow the validator and RPC data volumes without recreating t --values values.yaml ``` -3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows: +3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows. The loop echoes success or failure for each PVC—investigate any errors (insufficient quota, permissions, driver limits) before proceeding: ```bash # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` from step 1. @@ -153,8 +153,12 @@ Use this runbook to grow the validator and RPC data volumes without recreating t -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \ -o name \ | while read -r pvc; do - kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \ - -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}" + if kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \ + -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"; then + echo "Successfully patched ${pvc}" + else + echo "ERROR: Failed to patch ${pvc}" >&2 + fi done done ```