feat(operator): autogrow kopia cache PVC on persistent pressure#60
Merged
Conversation
Add a self-tuning loop for the kopia cache PVC: the restore Job's
pre-flight check evicts regenerable content when usage hits 85% and
POSTs to the operator's HTTP server; the operator bumps the PVC's
requested storage by 15% per event, capped at 2x the snapshot-derived
default. A chronically-pressured replica converges to a working PVC
size over a few restore cycles without manual intervention.
The signal path uses an HTTP callback (same pattern as the snapshot-list
and schema-migration jobs) rather than pod-log scraping — log fetching
has been unreliable in this operator across multiple iterations, the
callback path is what works here. New endpoint:
POST /api/v1/cache-pressure/{namespace}/{restore}
Pre-flight eviction in the Job script handles the immediate problem
(can't connect without cache cleanup if PVC is at 100%); the autogrow
handles the structural problem (working set legitimately needs more
than the snapshot-derived default). The two compose: prevent now,
fix permanently.
Bounded above by 2x the default size so a pathological case (e.g. a
leak in the pre-flight logic) can't grow the PVC indefinitely.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖
Summary
Self-tuning loop for the kopia cache PVC. Composes two halves:
Result: a chronically-pressured replica converges to a working PVC size over a handful of restore cycles. A one-off spike grows the PVC once and stops. No manual intervention.
Signal path
Uses an HTTP callback rather than pod-log scraping (the same pattern as the snapshot-list and schema-migration jobs already use).
```
POST /api/v1/cache-pressure/{namespace}/{restore}
```
Empty body — the path identifies which restore (and therefore which replica's cache PVC and which snapshot size) to bump. Handler in `bin/operator.rs` calls into `controllers::restore::grow_cache_pvc_after_pressure`.
Shape
Tests
Caveats