docs: add pvc resize runbook

[roderik]

Summary

• add StatefulSet PVC scaling runbook to README/README.tpl
• document helm values update and kubectl patch loop for resizing

Testing

• bun run check
• bun run typecheck
• bun test

Summary by Sourcery

Add a runbook section to the README and its template that guides users through scaling StatefulSet PVC storage without recreating pods.

Documentation:

• Add a new "Scale StatefulSet PVC storage (runbook)" section to README and README.tpl.
• Document steps to update Helm values, roll out changes with helm upgrade, patch PVCs using kubectl, and verify the resize.
• Include a note on enabling allowVolumeExpansion in the StorageClass if needed.

[gemini-code-assist]

Summary of Changes

Hello @roderik, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the project's documentation by introducing a critical runbook for scaling StatefulSet PVC storage. This new guide provides clear, step-by-step instructions for expanding data volumes for components like validators and RPC nodes, leveraging Helm for value updates and kubectl for direct PVC patching. The addition aims to streamline operations and prevent service interruptions when increasing storage capacity.

Highlights

• New Runbook for PVC Scaling: A comprehensive runbook has been added to the documentation, detailing the process for scaling StatefulSet Persistent Volume Claim (PVC) storage.
• Helm and Kubectl Usage: The runbook outlines how to update Helm values and use kubectl commands to patch existing PVCs, allowing for storage expansion without recreating StatefulSets.
• Support for Validator and RPC Volumes: The guide specifically addresses growing validator and RPC data volumes, ensuring continuous operation during storage expansion.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

To view in Slack, search for: 1758178299.154479

[sourcery-ai]

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `README.tpl:150-157` </location>
<code_context>
+   RELEASE="besu-network"
+   NAMESPACE="besu"
+
+   for component in validator rpc; do
+     kubectl get pvc -n "${NAMESPACE}" \
+       -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \
+       -o name \
+     | while read -r pvc; do
+         kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \
+           -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"
+       done
+   done
+   ```
</code_context>

<issue_to_address>
**suggestion:** Consider adding a note about handling failed PVC expansions.

Currently, failed PVC expansions are not reported. Consider implementing error handling or output monitoring to ensure all PVCs are resized as intended.

Suggested implementation:

```
   for component in validator rpc; do
     kubectl get pvc -n "${NAMESPACE}" \
       -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \
       -o name \
     | while read -r pvc; do
         if kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \
           -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"; then
           echo "Successfully patched ${pvc}"
         else
           echo "ERROR: Failed to patch ${pvc}" >&2
         fi
       done
   done
   ```

```

```
3. Expand the in-use PVCs with plain `kubectl` so the StatefulSets keep running while storage grows:

   > **Note:** The script below will print a message for each PVC indicating success or failure.  
   > If any PVC fails to expand, review the error output and investigate the cause (e.g., insufficient storage, permissions, or cluster limitations).

   ```bash
   NEW_SIZE="200Gi"
   RELEASE="besu-network"
   NAMESPACE="besu"

```
</issue_to_address>

### Comment 2
<location> `README.tpl:161-165` </location>
<code_context>
+4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize):
+
+   ```bash
+   kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w
+   ```
+
</code_context>

<issue_to_address>
**suggestion:** Consider specifying how long to wait for FileSystemResizePending to clear.

Including expected wait times or guidance for delayed resizing would help operators troubleshoot more effectively.

```suggestion
4. Confirm every claim reports the larger capacity (wait for `FileSystemResizePending` to clear if your CSI driver performs an in-pod resize):

   > **Note:** The `FileSystemResizePending` status typically clears within a few minutes, but may take up to 10–15 minutes depending on your storage backend and cluster load. If the status persists longer than expected, check your CSI driver logs and node status for issues. For troubleshooting, see [Kubernetes PVC resizing documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims).

   ```bash
   kubectl get pvc -n "${NAMESPACE}" -l app.kubernetes.io/instance="${RELEASE}" -w
   ```
```
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

suggestion: Consider adding a note about handling failed PVC expansions.

Currently, failed PVC expansions are not reported. Consider implementing error handling or output monitoring to ensure all PVCs are resized as intended.

Suggested implementation:

   for component in validator rpc; do
     kubectl get pvc -n "${NAMESPACE}" \
       -l app.kubernetes.io/instance="${RELEASE}",app.kubernetes.io/component="${component}" \
       -o name \
     | while read -r pvc; do
         if kubectl patch -n "${NAMESPACE}" "${pvc}" --type merge \
           -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"${NEW_SIZE}\"}}}}"; then
           echo "Successfully patched ${pvc}"
         else
           echo "ERROR: Failed to patch ${pvc}" >&2
         fi
       done
   done

3. Expand the in-use PVCs with plain kubectl so the StatefulSets keep running while storage grows:

   Note: The script below will print a message for each PVC indicating success or failure.
   If any PVC fails to expand, review the error output and investigate the cause (e.g., insufficient storage, permissions, or cluster limitations).

   NEW_SIZE="200Gi"
   RELEASE="besu-network"
   NAMESPACE="besu"
   

[gemini-code-assist]

Code Review

This pull request adds a runbook for resizing StatefulSet PVCs to README.md and README.tpl. The documentation is clear and the provided scripts for resizing are correct. I've suggested one minor improvement to make the runbook less error-prone by adding a comment to remind users to synchronize the size value between the Helm configuration and the patch script.

[gemini-code-assist]

The NEW_SIZE variable is hardcoded here and also needs to be set in the Helm values in step 1. This can be error-prone if a user forgets to update it in both places. I suggest adding a comment to remind the user to keep these values in sync, improving the runbook's robustness.

   # IMPORTANT: Set this to the same value as `network-nodes.persistence.size` in your Helm values from step 1.
   NEW_SIZE="200Gi"