Skip to content

feat(operator): autogrow kopia cache PVC on persistent pressure#60

Merged
passcod merged 1 commit into
mainfrom
operator-kopia-cache-autogrow
Jun 5, 2026
Merged

feat(operator): autogrow kopia cache PVC on persistent pressure#60
passcod merged 1 commit into
mainfrom
operator-kopia-cache-autogrow

Conversation

@passcod

@passcod passcod commented Jun 5, 2026

Copy link
Copy Markdown
Member

🤖

Summary

Self-tuning loop for the kopia cache PVC. Composes two halves:

  1. Pre-flight (in the restore Job): if cache PVC usage ≥85% before kopia starts, evict the regenerable content (`/tmp/kopia/cache` and `/tmp/kopia/logs/content-logs`). Prevents the "kopia can't write its 4KB config" failure mode that triggers when the PVC hits 100%.
  2. Autogrow (in the operator): when the pre-flight eviction fires, the Job POSTs to a new operator endpoint, and the operator bumps the cache PVC's requested storage by 15%. Capped at 2× the snapshot-derived default.

Result: a chronically-pressured replica converges to a working PVC size over a handful of restore cycles. A one-off spike grows the PVC once and stops. No manual intervention.

Signal path

Uses an HTTP callback rather than pod-log scraping (the same pattern as the snapshot-list and schema-migration jobs already use).

```
POST /api/v1/cache-pressure/{namespace}/{restore}
```

Empty body — the path identifies which restore (and therefore which replica's cache PVC and which snapshot size) to bump. Handler in `bin/operator.rs` calls into `controllers::restore::grow_cache_pvc_after_pressure`.

Shape

  • New helpers in `controllers/restore/builders.rs`:
    • `KOPIA_CACHE_PRESSURE_GROWTH_FACTOR = 1.15`
    • `kopia_cache_pvc_max(snapshot)` — 2× the default size, the growth cap.
    • `next_cache_pvc_size_after_pressure(current, snapshot)` — current × 1.15, capped.
  • New `grow_cache_pvc_after_pressure` in `controllers/restore.rs` — patches the PVC's `spec.resources.requests.storage`. Best-effort; logs and continues on failure.
  • New `Context::cache_pressure_callback_url` and `/api/v1/cache-pressure/{namespace}/{restore}` route.
  • `build_restore_job` takes the callback URL and sets it as `CACHE_PRESSURE_CALLBACK_URL` on the restore container; the embedded script POSTs to it when pre-flight evicts.

Tests

  • `next_cache_pvc_size_after_pressure_bumps_and_caps` — verifies bumps grow then saturate at the 2× cap.
  • `restore_job_passes_cache_caps_and_log_rotation` extended to require the new env var and the script's POST step.

Caveats

  • Growth depends on the storage class supporting volume expansion (we already rely on this for the snapshot-growth ratchet). Failure is logged and the restore proceeds with the existing size.
  • Growth is permanent (never shrinks), matching the existing PVC-resize pattern.

Add a self-tuning loop for the kopia cache PVC: the restore Job's
pre-flight check evicts regenerable content when usage hits 85% and
POSTs to the operator's HTTP server; the operator bumps the PVC's
requested storage by 15% per event, capped at 2x the snapshot-derived
default. A chronically-pressured replica converges to a working PVC
size over a few restore cycles without manual intervention.

The signal path uses an HTTP callback (same pattern as the snapshot-list
and schema-migration jobs) rather than pod-log scraping — log fetching
has been unreliable in this operator across multiple iterations, the
callback path is what works here. New endpoint:
  POST /api/v1/cache-pressure/{namespace}/{restore}

Pre-flight eviction in the Job script handles the immediate problem
(can't connect without cache cleanup if PVC is at 100%); the autogrow
handles the structural problem (working set legitimately needs more
than the snapshot-derived default). The two compose: prevent now,
fix permanently.

Bounded above by 2x the default size so a pathological case (e.g. a
leak in the pre-flight logic) can't grow the PVC indefinitely.
@passcod passcod enabled auto-merge June 5, 2026 11:24
@passcod passcod merged commit 0d6f4ef into main Jun 5, 2026
18 checks passed
@passcod passcod deleted the operator-kopia-cache-autogrow branch June 5, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant