Skip to content

fix(dp): requeue volume populator wait states#10458

Closed
weicao wants to merge 6 commits into
mainfrom
bugfix/volume-populator-conflict-requeue-on-10447
Closed

fix(dp): requeue volume populator wait states#10458
weicao wants to merge 6 commits into
mainfrom
bugfix/volume-populator-conflict-requeue-on-10447

Conversation

@weicao

@weicao weicao commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

This is a stacked follow-up on PR #10447.

It fixes two generic VolumePopulator retry/requeue gaps found during etcd restore validation:

  • return conflict errors from handleSyncPVCError even when the target PVC already has the Populating condition, so controller-runtime can retry stale status/resource-version updates
  • actively requeue Populate() while prepareData jobs are not finished, so the controller does not depend on a later external PVC event after the restore container has exited and the restore-manager sidecar is still running

It also carries two changes beyond the retry/requeue scope (called out explicitly for review):

  • decidePVCRestore: when no volume-level restore exists and the full target selector rejected the PVC (pod-only labels), clear skipPostReady if the PVC's component matches the target's component selector, so the target component's PVCs still drive postReady for logical backups. Note the overlap with fix: match backup targets to restore PVC labels #10354, which fixes the selector matching itself — once fix: match backup targets to restore PVC labels #10354 merges, this fallback should become a no-op for the logical-backup case and can be re-evaluated.
  • patchInternalRestoreStatus / patchInternalRestoreMetaAndStatus: re-read the latest Restore and apply only the fields this reconcile intended to change (original→desired delta onto latest), instead of merge-patching from a possibly stale base, to avoid clobbering concurrent Restore status writers.

Validation

Focused unit tests:

go test ./controllers/dataprotection -run 'TestPopulateRequeuesWhenRestoreContainerFinishedBeforeJobComplete|TestHandleSyncPVCError(RequeuesConflictWhenPVCIsPopulating|KeepsInternalRequeueWhenPVCIsPopulating)|TestPatchInternalRestoreStatusUsesLatestRestoreResourceVersion' -count=1

Runtime validation on stacked head 943044aed667a71c2848da47c7b103920d6f6245:

  • scope: idc4 node4 single-node vcluster, KB 1.2.0-alpha.2, local-sideload DP image
  • image: kubeblocks-dataprotection:pr-10447-943044aed-20260626
  • image identity: sha256:6744bc202b83d4e4314534adeb9a178307090334f0f00ddf1aef5538860fbf65
  • result: Gate 4 v3 accepted scoped PASS
  • restore: fully automatic, no manual PVC patch, total restore time 41s, prepareData completed in 13s
  • readback: 3/3 etcd keys, revision=4
  • evidence: build 9/9, import 3/3, runtime 11/11, cleanup 26/26 SHA checks OK

Scope boundary

The runtime PASS above was run on the stacked combined head 943044aed, not on PR #10447 head alone.
PR #10447 should remain the original restoreData/skipPostReady selector fix; this PR carries the follow-up generic VolumePopulator retry/requeue fixes.

weicao added 6 commits June 24, 2026 00:57
When a backup ActionSet has only postReady (no prepareData/targetVolumes),
decidePVCRestore incorrectly propagated skipPostReady=true for PVCs whose
component did not match the backup target. This blocked the sole restore
path for multi-component architectures like TiDB where the backup targets
the SQL frontend but data lives in PD/TiKV PVCs.

Override skipPostReady to false when restoreData is false: if there is no
volume-level restore, PVC-selector mismatch should not block postReady.

Fixes #10418
Address leon-ape's review: the previous fix `if !restoreData { skipPostReady = false }`
was too broad — in multi-component scenarios (e.g. TiDB with pd/tikv), non-target
component PVCs would also get skipPostReady cleared, potentially executing PostReady
restore on wrong components.

Refined approach: when !restoreData && skipPostReady, check if the PVC's component
label matches the backup target component. Only clear skipPostReady (and recover
sourceTarget) when the PVC belongs to the backup target component. Non-target
component PVCs keep skipPostReady=true.
… label

When the backup target PodSelector exists but has no KBAppComponentLabelKey,
targetMatchesComponent now returns false instead of true. This prevents the
fallback from matching arbitrary component PVCs when we cannot confirm
component membership.

The no-PodSelector case (nil selector) still returns true -- that means
the backup applies to all components by design.
@weicao weicao requested review from a team, ldming and wangyelei as code owners June 25, 2026 21:44
@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.11765% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.05%. Comparing base (4933980) to head (943044a).
⚠️ Report is 17 commits behind head on main.

Files with missing lines Patch % Lines
...llers/dataprotection/volumepopulator_controller.go 74.11% 15 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10458      +/-   ##
==========================================
+ Coverage   62.04%   62.05%   +0.01%     
==========================================
  Files         533      533              
  Lines       63625    63707      +82     
==========================================
+ Hits        39473    39535      +62     
- Misses      20550    20565      +15     
- Partials     3602     3607       +5     
Flag Coverage Δ
unittests 62.05% <74.11%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weicao weicao marked this pull request as draft June 26, 2026 03:29
@weicao weicao removed request for a team, ldming and wangyelei June 26, 2026 03:29
@weicao weicao force-pushed the bugfix/skip-postready-multi-component branch from 22180f7 to 806bc35 Compare June 29, 2026 15:35
Base automatically changed from bugfix/skip-postready-multi-component to main July 2, 2026 14:31
@github-actions github-actions Bot added the size/L Denotes a PR that changes 100-499 lines. label Jul 2, 2026
@weicao

weicao commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

Self-review outcome: the core change here is still the 1s requeue polling that was explicitly rejected twice on this same wait state (#10477, #10481 — the direction given was event-driven Pod watch + Job/PVC label mapping), and the by-design question raised by @wangyelei on #10477 (stop-restore-manager=true is only set after all restore job containers complete) has not been answered with new evidence. This PR also bundles four changes; per the one-owner-investigation direction given when #10481 was closed, this should not stay open as-is. The one independently valuable piece is the conflict-requeue fix in handleSyncPVCError (the current code swallows conflicts for PVCs that already carry a Populating condition and returns Reconciled) — that can be extracted into a narrow PR with its own reproduction. Recommending close + extract.

@weicao

weicao commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

Closing per the analysis above: the polling approach was rejected twice on this wait state (#10477/#10481) and the by-design question remains unanswered. The conflict-requeue piece of handleSyncPVCError will be extracted into a narrow PR with its own reproduction if it holds up independently.

@weicao weicao closed this Jul 5, 2026
@github-actions github-actions Bot added this to the Release 1.2.0 milestone Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L Denotes a PR that changes 100-499 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant