fix: etcd fails due to unaccounted space usage (e.g. WAL) by cwrau · Pull Request #115 · teutonet/cluster-api-provider-hosted-control-plane

cwrau · 2026-03-31T13:17:30Z

This way the real volume usage is looked at and used for resizing.

gemini-code-assist

Code Review

This pull request introduces a new EtcdVolumeStatsProvider to monitor and reconcile etcd volume usage by querying kubelet stats, allowing for more accurate tracking of disk usage beyond just the etcd database size. It includes updates to the etcd client to support filtering by ready pods, improvements to the etcd cluster reconciler to incorporate filesystem usage metrics, and necessary dependency updates. I have no feedback to provide.

Copilot

Pull request overview

This PR fixes etcd volume auto-resize logic by using actual filesystem usage (via kubelet stats summary) instead of relying solely on etcd-reported DB size, accounting for extra space usage like WAL/snapshot files.

Changes:

Add an etcd volume stats provider that queries kubelet /stats/summary and computes max etcd volume usage across pods.
Update the etcd cluster reconciler to compute an “effective” volume usage as max(dbSize, filesystemUsage) and emit warnings when they diverge significantly.
Update etcd status collection to target ready pods and parallelize member calls; adjust wiring, stubs, and tests accordingly.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`test/etcd_stubs.go`	Updates etcd client stub signature and adds a stub for the new volume stats provider.
`pkg/reconcilers/etcd_cluster/volume_stats/volume_stats.go`	Implements kubelet-based filesystem usage collection for etcd data volumes.
`pkg/reconcilers/etcd_cluster/volume_stats/volume_stats_test.go`	Adds unit tests for volume identification (`isEtcdDataVolume`).
`pkg/reconcilers/etcd_cluster/reconciler.go`	Integrates filesystem usage into resize decisions; adds pod listing and adjusts health check placement and args gating.
`pkg/reconcilers/etcd_cluster/reconciler_test.go`	Extends tests for new effective-usage and warning behaviors.
`pkg/reconcilers/etcd_cluster/etcd_client/etcd_client.go`	Changes `GetStatuses` API to accept ready pod names and parallelizes per-member calls.
`pkg/hostedcontrolplane/controller.go`	Wires the new volume stats provider into the reconciler; adjusts events RBAC verbs.
`go.mod`	Bumps several dependencies (k8s libs, controller-runtime, grpc, cilium).
`go.sum`	Updates checksums for dependency upgrades.
`.golangci.yaml`	Adds import aliasing for kubelet stats API package.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 33 out of 35 changed files in this pull request and generated 1 comment.

This way the real volume usage is looked at and used for resizing.

chore: replace span/log/recorder with emit

Copilot

Pull request overview

Copilot reviewed 33 out of 35 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 33 out of 35 changed files in this pull request and generated 1 comment.

Copilot AI review requested due to automatic review settings March 31, 2026 13:17

cwrau enabled auto-merge March 31, 2026 13:17

Copilot started reviewing on behalf of cwrau March 31, 2026 13:18 View session

gemini-code-assist Bot reviewed Mar 31, 2026

View reviewed changes

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Copilot started work on behalf of cwrau March 31, 2026 13:25 View session

Copilot AI review requested due to automatic review settings March 31, 2026 13:28

auto-merge was automatically disabled March 31, 2026 13:28
Head branch was pushed to by a user without write access

cwrau review requested due to automatic review settings March 31, 2026 13:28

Copilot finished work on behalf of cwrau March 31, 2026 13:29

Copilot AI review requested due to automatic review settings March 31, 2026 13:30

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from ea81376 to 2e6eac4 Compare March 31, 2026 13:30

Copilot started reviewing on behalf of cwrau March 31, 2026 13:31 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Comment thread go.mod

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from 2e6eac4 to 516cda8 Compare March 31, 2026 13:37

Copilot AI review requested due to automatic review settings March 31, 2026 13:39

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from 516cda8 to b37d3d4 Compare March 31, 2026 13:39

Copilot started reviewing on behalf of cwrau March 31, 2026 13:40 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go Outdated

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go Outdated

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go Outdated

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from b37d3d4 to 2a5b8ad Compare April 1, 2026 14:06

cwrau enabled auto-merge April 1, 2026 14:07

Copilot AI review requested due to automatic review settings April 17, 2026 08:17

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from 2a5b8ad to 3c927cd Compare April 17, 2026 08:17

Copilot started reviewing on behalf of cwrau April 17, 2026 08:17 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from 3c927cd to d38cd45 Compare April 17, 2026 09:09

Copilot AI review requested due to automatic review settings April 20, 2026 09:38

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from d38cd45 to 1dc1b75 Compare April 20, 2026 09:38

Copilot started reviewing on behalf of cwrau April 20, 2026 09:38 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread pkg/reconcilers/etcd_cluster/etcd_client/etcd_client.go

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go Outdated

Comment thread go.mod

Copilot AI review requested due to automatic review settings April 30, 2026 10:13

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from 382c399 to 76e651c Compare April 30, 2026 10:13

Copilot started reviewing on behalf of cwrau April 30, 2026 10:14 View session

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Comment thread pkg/reconcilers/etcd_cluster/volume_stats/volume_stats_test.go Outdated

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go Outdated

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go

Comment thread go.mod

Copilot AI review requested due to automatic review settings May 18, 2026 08:20

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from fe0dd96 to b4747de Compare May 18, 2026 08:20

Copilot started reviewing on behalf of cwrau May 18, 2026 08:20 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

Comment thread test/context.go Outdated

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from b4747de to daeb324 Compare May 18, 2026 09:02

marvinWolff previously approved these changes May 18, 2026

View reviewed changes

cwrau added this pull request to the merge queue May 18, 2026

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch May 18, 2026

cwrau added 3 commits May 18, 2026 17:07

fix: etcd fails due to unaccounted space usage (e.g. WAL)

15ec7e1

This way the real volume usage is looked at and used for resizing.

fix: inverted condition

04e60c2

chore: adjust etcd space calculation and NOSPACE alarm dismissal

8e3d4cd

chore: replace span/log/recorder with emit

Copilot AI review requested due to automatic review settings May 18, 2026 15:16

cwrau dismissed marvinWolff’s stale review via 8e3d4cd May 18, 2026 15:16

cwrau force-pushed the fix/etcd-unaccounted-space-usage branch from daeb324 to 8e3d4cd Compare May 18, 2026 15:16

cwrau enabled auto-merge May 18, 2026 15:16

Copilot started reviewing on behalf of cwrau May 18, 2026 15:16 View session

Copilot AI reviewed May 18, 2026

View reviewed changes

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go

Comment thread pkg/reconcilers/etcd_cluster/reconciler_test.go Outdated

cwrau added 2 commits May 19, 2026 10:34

fix: import cycle

b799065

chore: remove duplicate reconcileETCDSpaceUsage call

1a7c0a6

Copilot AI review requested due to automatic review settings May 19, 2026 08:45

Copilot started reviewing on behalf of cwrau May 19, 2026 08:46 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

Comment thread pkg/reconcilers/etcd_cluster/reconciler.go Outdated

fix: use ParseTolerant

2688d8d

marvinWolff approved these changes May 19, 2026

View reviewed changes

cwrau added this pull request to the merge queue May 19, 2026

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch May 19, 2026

Conversation

cwrau commented Mar 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants