feat: add the EtcdMirror replication engine library (pkg/mirroragent)

[xrl]

Adds pkg/mirroragent: the EtcdMirror replication engine as a standalone library with zero Kubernetes dependency — pure etcd clientv3 against a Config, consumed by the agent binary and controller in the next PRs of the series. Builds on the EtcdMirror API PR (types only); this PR adds no controller or workload wiring.

Depends on: #406 (EtcdMirror CRD/API types — the previous rung of this series).

Architecture

Unpinned chunked scan + watch-replay from R0 (the reflector pattern), rather than mirror.Syncer. Why mid-scan compaction cannot wedge the engine:

1. At scan start one linearizable Get (WithCountOnly) returns Header.Revision = R0 and the total count in a single RPC.
2. The watch opens at R0+1 before any scan page is read. R0 was observed this instant by a linearizable read, so it cannot already be compacted.
3. Every scan page is an unpinned Get (no WithRev). ErrCompacted is a property of reads pinned below the compact revision; an unpinned read is immune by construction, at every point during the scan, regardless of concurrent compactions. The failure class is removed, not detected-and-retried.
4. The only remaining compaction hazard is the watch stream itself going quiet long enough that a re-Watch(WithRev(watermark+1)) lands below the compact revision — identical in shape during InitialSync and steady state, handled by one mechanism: forced resync with a mandatory mark-and-sweep prune (durable across crashes via a prune-pending flag in the checkpoint).

Scan and watch are not sequential phases with a handoff race: the watch is live for the entire scan, and scan writes may interleave with replayed watch writes because both write the same idempotent final value for a given key — convergence to the last-write value; a duplicate Put of identical content is a correctness no-op.

Fence protocol

The checkpoint/watermark is stored in the target etcd at a reserved key (\x00-after-prefix convention; exact-match excluded from scans, counts, prune and RequireEmpty), written in the same Txn as each applied batch. All safety rests on one discipline, identical on every write path (apply, reconcile repair, prune delete, cutover role-flip):

If(Compare(ModRevision(fenceKey), "=", observedModRev)).
Then(dataOps..., Put(fenceKey, next))
// on !Succeeded: re-read, recompute, retry — never blind re-Commit

etcd's Compare has no JSON-field predicates, so linkUID/epoch/role are payload for humans and the engine's state machine, never comparison predicates. Cutover safety falls out for free: the role-flip Txn bumps the fence's ModRevision, so any writer holding a pre-flip observedModRev fails its next compare loudly — no special role-check code. An undecodable checkpoint fails closed (permanent, operator inspects and deletes the reserved key): a corrupt fence proves nothing about ownership, so no write — least of all a resync's prune — is provably safe.

Retry ownership: clientv3 auto-retries Gets only on codes.Unavailable; Txn/Put/Delete are write-at-most-once. The engine owns 100% of write-path retry/backoff, and a fenced-Txn retry after an ambiguous timeout re-reads the fence first (the Txn's own success bumped the fence ModRevision).

Bounded memory, batching, error taxonomy

• Pull-based single-page scan with byte-bounded pages (adaptive per-page limit — the Range API has no byte-limit primitive); watch replay buffer bounded by watchBufferBytes, on overflow the scan restarts from a fresh R0 (a bounded retry, not unbounded growth).
• Batches flush only at source-revision boundaries (whole revisions coalesced, never split); one MaxTxnOps slot reserved for the checkpoint write.
• codes.ResourceExhausted is disambiguated three ways — ErrNoSpace → quota (park until operator acts), ErrTooManyRequests → throttle (distinct backoff curve), client send-cap → permanent — never classified by gRPC code alone. Keys surfaced in errors are redacted (spec-public prefix + sha256 stub); values never surface. An N-consecutive-forced-resync livelock detector catches retention-vs-throughput misconfiguration (ResyncLoopDetected).
• Liveness: watermark-derived progress via WithProgressNotify + WithRequireLeader + client-driven RequestProgress, gRPC keepalives on both clients, per-unary-RPC deadlines (watch excluded). A version probe at connect enforces the ≥3.4 source floor and gates progress-notify trust on 3.4.25/3.5.8.

Tests

Unit tests per module (fence encode/decode + corruption fail-closed, error classification table incl. the three-way ResourceExhausted split, batch revision-boundary/flush accounting, rewrite formula, config validation, backoff curve mapping, key redaction) plus an embedded-etcd integration suite: cold start, live tail, mid-scan compaction, restart resume, fence overlap between two agents, role-flip cutover fencing, oversized revisions, all three initialSync modes, drain, target quota exhaustion, watch-buffer overflow restart, prune-pending crash-resume, cluster-ID-mismatch re-arm, resync-loop detection, excluded prefixes, corrupt/future-version checkpoints, version floor, progress-notify metadata regression, and idempotent replay.

Verified with make test and make verify.

---

PR series — operability fixes, TLS & EtcdMirror

Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.

	PR	Lands	After
	Reviewable now — small, any order
🟢	#374	Requeue instead of swallowing the client-cert provisioning error	—
🟢	#391	Grant events.k8s.io RBAC so operator Events are actually recorded	—
🟢	#392	Correlate members[]/leaderID from one health snapshot (consistent leader)	—
🟢	#393	Propagate user-supplied altNames.ipAddresses into certificates	—
🟢	#394	Accept day-suffix validityDuration (365d, 100d12h) as documented	—
🟢	#395	Surface early reconcile errors as a Degraded condition (was empty status)	—
🟢	#379	Configurable reconcile worker pool (--max-concurrent-reconciles)	—
🟢	#369	kind-based stress/scale e2e harness (1/3/7 members, churn, quorum watcher)	—
🟢	#398	podTemplate.spec scheduling fields: topologySpread, resources, priorityClass, schedulerName	—
🟢	#397	First-class Helm chart covering the full config/ surface	—
🟢	#399	Quorum-aware PodDisruptionBudget per EtcdCluster	—
🟢	#400	Scale-in removes the right member and transfers leadership first	—
🟢	#401	Backend quota + auto-compaction spec fields, NOSPACE alarm surfacing	—
🟢	#402	Reconcile-loop gates replace the blocking StatefulSet-ready wait	—
🟢	#403	Quorum-gated one-pod-at-a-time version upgrades via OnDelete	—
	TLS stack — in order
🟢	#376	Independent spec.tls.{peer,client} surfaces (breaking alpha API)	—
⚪	#377	TLSReady condition + TLS lifecycle Events	#376
⚪	#378	Multi-member TLS quorum e2e + PeerCANotShared	#377
	EtcdMirror — in order
🟢	#406	EtcdMirror CRD: one-way cross-cluster replication API (types + CEL)	—
🟢→	#407	pkg/mirroragent replication engine (fenced checkpoint-in-target)	#406
🟢	#408	Periodic reconciliation pass wired into the engine's steady-state loop	#407
🟢	#412	mirror-agent binary: config/TLS-reload/statusz/metrics + kind e2e	#408
🟢	#413	EtcdMirror controller: Deployment rendering, statusz-driven conditions, guards, fenced finalizer	#412
🟢	#414	CRD/sample/editor+viewer-role kustomize wiring	#413
🟢	#415	Controller-side phase/condition gauges, shipped PrometheusRule + dashboard	#414
🟢	#416	Full-lifecycle kind e2e: fence overlap, compaction+prune, restart resume, cert rotation, cutover/reversal	#415
	Parked as drafts pending #363
⚪	#382	Per-cluster domain metrics on the operator /metrics endpoint	—
⚪	#384	EtcdCluster admission webhooks (consolidating with #328)	#363
⚪	#386	EtcdBackup CR → object storage (S3/GCS)	#363
⚪	#387	Automatic quorum-loss disaster recovery (bootstrap-latch guarded)	#363

🟢 ready · ⚪ draft

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.