feat: add the mirror-agent binary for EtcdMirror (config, TLS reload, statusz/metrics, kind e2e)#412
Open
xrl wants to merge 10 commits into
Open
feat: add the mirror-agent binary for EtcdMirror (config, TLS reload, statusz/metrics, kind e2e)#412xrl wants to merge 10 commits into
xrl wants to merge 10 commits into
Conversation
Introduces the EtcdMirror CRD (schema only, no controller/agent) that will describe a continuous one-way key-range sync from a source etcd cluster to a target etcd cluster. Adds EtcdMirrorSpec/Status, the EtcdMirrorEndpoint/TLS/Auth/Sync/Checkpoint/Reconciliation sub-specs, phase and condition constants (including TargetThrottled and ReplicationLagExceeded), printer columns, and CEL XValidation rules for the destPrefix/noDestPrefix mutual exclusion, the endpointList/serviceRef oneOf, and the insecureSkipVerify/ insecureSkipVerifyAcknowledgeRisk companion-field requirement. Registers EtcdMirror/EtcdMirrorList in the scheme, regenerates zz_generated.deepcopy.go and the CRD manifest via controller-gen, and adds a realistic sample CR. CEL rules are covered by a table-driven envtest suite in etcdmirror_cel_test.go, following the tls_cel_test.go pattern, exercised against a real envtest apiserver. The design's Metrics field (reusing EtcdClusterSpec's MetricsSpec) is intentionally omitted: MetricsSpec does not exist on this branch yet and will land with pr/domain-metrics; it will be added in a later PR once that type is importable. No controller or agent logic in this PR -- types, generated code, and CRD schema only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Xavier Lange <xrlange@gmail.com>
Reshape the EtcdMirror API for the Design-3 engine (checkpoint stored in
the target etcd, stateless agent) per the 48-finding cross-cloud review.
Deletions:
- checkpoint.storageSpec (no PVC; checkpoint lives in the target at a
reserved fenced key)
- expectEmptyPrefix (superseded by initialSync.mode)
- noDestPrefix (redundant; one anchored rewrite formula
key' = target.prefix + destPrefix + TrimPrefix(key, source.prefix))
- syncInterval
Additions/renames:
- spec.mode: Sync|Drain; status.cutover{drainTargetRevision,
drainedRevision, verifiedTime, counts, leasedKeyCount}
- initialSync.mode: RequireEmpty|Overwrite|OverwriteAndPrune,
startRevision
- sync: requestTimeout, excludePrefixes, pageKeyLimit,
watchBufferBytes; batching flushes only at source-revision boundaries
- TLS: secretRef pointerized (nil = system trust roots), caBundleRef
- pod resources; configurable reserved checkpoint key
- watermark-derived lag/liveness semantics; frozen condition vocabulary
(TargetQuotaExhausted, ResyncLoopDetected, CutoverReady,
InvariantsHeld, LearnerEndpoint, Prefix/DirectionConflict) and Event
Reason constants as API contract
- status: source/target versions and cluster IDs, leaseBackedKeyCount,
always-on source/target key counts, initialSyncTotalKeyCount
- CEL: scheme-vs-TLS rules, prefix/rewrite immutability
- docs/etcdmirror.md (fidelity caveats, one-way contract) and
regenerated API reference
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
Purpose-built replication engine, zero k8s dependency; replaces the
clientv3 mirror.Syncer plan.
Architecture: unpinned chunked scan + watch-replay from R0 (reflector
pattern) — the watch starts at the revision observed before the scan
and events are buffered/replayed over the scanned base, eliminating
mid-scan compaction as a failure class. Pull-based single-page scan
with byte-bounded pages keeps agent memory bounded; the source watch
is cancelled on sustained target backoff and resumed from checkpoint.
Fence protocol: checkpoint/watermark stored in the target etcd at a
reserved key (\x00-after-prefix convention, exact-match excluded from
scans/counts/prune/RequireEmpty), written in the same Txn as each
applied batch and guarded by a mod_revision compare on every write
path (applies, reconciliation repairs, prune deletes). Fence value
carries {linkUID, epoch, role}; role flips to Primary at cutover so
straggler applies fail their compare loudly. Undecodable checkpoints
fail closed; prune-pending is durable across restarts.
Batching: flush only at source-revision boundaries (whole revisions
coalesced, never split), ~1MiB byte watermark, one MaxTxnOps slot
reserved for the checkpoint write.
Error taxonomy: ErrCompacted classified distinctly;
rpctypes.ErrNoSpace -> TargetQuotaExhausted (permanent until operator
acts); oversized-Txn InvalidArgument and >2MiB client send-cap
ResourceExhausted both permanent with the offending key surfaced
redacted; throttle-distinct backoff; N-consecutive-forced-resync
livelock detector.
Liveness: watermark-derived progress via WithProgressNotify +
WithRequireLeader, client-driven RequestProgress, gRPC keepalives on
both clients, per-unary-RPC deadlines (watch excluded). Version probe
at connect enforces the >=3.4 source floor and gates progress-notify
trust on 3.4.25/3.5.8. Anchored prefix rewrite via the single
documented formula. Lease-backed keys detected and counted
(kv.Lease != 0).
Tests: unit coverage per module plus an embedded-etcd integration
suite covering scan/tail convergence, fence overlap and epoch/role
rejection, compaction during scan and drain, forced-resync
mark-and-sweep, checkpoint resume, prune, and error classification.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
A single post-heal put can race the recovery attempt's restart backoff and land below the scan's R0: the scan applies it, the live tail has nothing to deliver, and no live apply ever clears the latch. Write a fresh key per poll tick so one is guaranteed to arrive via the tail. Signed-off-by: Xavier Lange <xrlange@gmail.com>
Deadline-driven pass on the steady-state tail loop: consume arms a timer from a Run-goroutine-owned deadline and runs the diff-and-repair pass inline, so it can never overlap the genesis scan, a forced-resync sweep (which re-arms the deadline — it just produced the same signal), a drain (whose own verification supersedes it), or itself. The drain repair branch now records its drift, recordKeyCounts stamps the pass-completion time on every counts-producing pass — including the count-only drain verify — keeping the always-on counts contract behind InvariantsHeld, and Validate rejects negative intervals. Signed-off-by: Xavier Lange <xrlange@gmail.com>
- publish counts, pass timestamp, and drift in one Snapshot update inside reconcilePass; drop the torn caller-side recordDrift - cancel the live source watch once a periodic/drain repair pass queues more than a few seconds of MaxOpsPerSecond pacing, bounding clientv3's unbounded per-watcher buffer during a long healthy pass - correct the ReconcileInterval re-arm, key-count population, and maybeReconcile error-flow comments to match actual behavior - damage the target atomically (single Txn) in the drift test so a pass boundary cannot split the damages across two drifts Signed-off-by: Xavier Lange <xrlange@gmail.com>
…pshot JSON-tag Snapshot/CutoverStatus/Drift (lowerCamel, omitzero on times) as the /statusz wire contract, and add two additive fields the agent binary's metrics need: KeysAppliedTotal (data ops committed in fenced Txns) and ForcedResyncCountByReason (labeled-counter split of ForcedResyncCount). No engine semantics change; no field renames. Signed-off-by: Xavier Lange <xrlange@gmail.com>
cmd/mirror-agent wires pkg/mirroragent to real config, TLS and auth:
- Flags-only config (operator convention); secrets arrive as mounted-file
paths, never flag values or env. Quantities parse as resource.Quantity;
zero/unset selects engine defaults; the reconciliation enabled+interval
-> ReconcileInterval translation (1h default) lands here as assigned.
- TLS per side via transport.TLSInfo file paths: the client leaf pair is
re-read on every handshake, so certificate rotation needs no restart;
--<side>-ca-bundle-file wins over --<side>-ca-file (caBundleRef
precedence). Schemeless endpoints get the scheme implied by
--<side>-tls; a contradicting scheme is an error (the CEL rules'
binary mirror). RBAC auth reads username/password files once; the
token identity wins over the certificate CN.
- One listener serves /statusz (JSON of the tagged mirroragent.Snapshot
- the wire contract the controller decodes), /healthz (always 200),
/readyz (ready = past Connecting and not Failed; Degraded and Drained
are ready), /metrics (standalone Prometheus registry;
etcd_mirror_agent_* incl. tls_cert_expiry_timestamp_seconds{side,kind}
refreshed from the mounted files every 5m).
- On terminal engine states (drained / permanent failure) the process
lingers serving status instead of exiting: a crashloop would hide the
terminal state and re-run genesis under the same epoch. Exit 0 = clean
signal shutdown, 1 = startup or permanent engine failure.
- Packaging: both binaries in the one existing image (ENTRYPOINT stays
/manager; the rendered Deployment overrides command).
Signed-off-by: Xavier Lange <xrlange@gmail.com>
- Pre-initialize forced_resync_total at 0 for every known reason so rate()/increase() can see the first resync (a counter's first sample yields no delta); unknown future reasons still pass through. - Serialize Snapshot cluster IDs as JSON strings: random uint64s exceed 2^53 and round through float64-based consumers (jq, JavaScript), matching the checkpoint's existing convention. - Reject TLS material flags (cert/key/ca/bundle/server-name/ insecure-skip-verify) without --<side>-tls instead of silently dialing cleartext; gate the cert-expiry gauge on tls too. - Fail startup on empty credential files: clientv3 silently skips authentication unless both username and password are non-empty. - Don't log "engine failed permanently" (or linger) when Run's return raced a shutdown signal; classify cancellation once in one place. - Restore default signal disposition after the first signal so a second SIGINT/SIGTERM kills a wedged shutdown, like the manager's ctrl.SetupSignalHandler. - Make the live test's timeout diagnostic read the snapshot after polling (require.Eventually copies msgAndArgs before it). - Drop PR-ladder "rung" vocabulary from doc comments. Signed-off-by: Xavier Lange <xrlange@gmail.com>
Two size-1 EtcdClusters plus a mirror-agent pod wired source->target in a dedicated namespace. Proves put/delete replication within a polled 10s ceiling (250ms interval, convergence time logged) and that /statusz (keysAppliedTotal, watermark, lastProgressTime) and the /metrics keys_applied_total counter advance past baseline. The agent image is distroless, so statusz/metrics are reached via the API-server pod proxy. Signed-off-by: Xavier Lange <xrlange@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xrl The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @xrl. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This was referenced Jul 7, 2026
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
cmd/mirror-agent: the standalone binary that wires thepkg/mirroragentreplication engine (#407) to real configuration, TLS, and auth, and exposes the agent's observability surface. The EtcdMirror controller (next PR in the series) renders a Deployment running this binary; its flag surface is that contract. Zero Kubernetes API dependency — the binary never talks to the apiserver.Depends on: #408 (periodic reconciliation — previous rung of this series).
Configuration
Flags only, matching the repo's manager-binary conventions; secrets arrive as mounted-file paths (
--source-username-file,--source-cert-file, …), never as flag values or env. Flag semantics mirror the CRD field docs exactly: endpoint scheme normalization follows the same rules as the API's CEL validation,--watch-buffer-bytesstays in lockstep with the engine default, and--reconcile-enabledwithout an interval translates to the documented 1h default. Empty credential files fail startup rather than silently disabling auth, and setting any TLS material flag without--<side>-tlsis a startup error rather than a silent cleartext downgrade.TLS and auth
Client TLS is built from file paths via
go.etcd.io/etcd/client/pkg/v3/transport.TLSInfo, so the leaf cert/key are re-read from the mount on every handshake — the rotation contract pinned in the API docs ("rotation requires no pod restart"). A separate--<side>-ca-bundle-filetakes precedence over the identity secret'sca.crt(mirroringtls.caBundleRef); an empty TLS config means server-auth against system roots. Username/password auth is per side; when both a token identity and a client-cert CN are present the token identity wins, matching the documented CRD contract.Observability
/statusz— the engineSnapshotserialized through its JSON tags; this is the wire contract the controller decodes (same Go type, no shadow struct to drift). Includes watermark, per-side cluster IDs (string-encoded — uint64s don't survive float64 JSON consumers) and versions, initial-sync progress with denominator,keysAppliedTotal, forced-resync counts by reason, key counts, the cutover block, and the last scan-restart cause./metrics— a standalone Prometheus registry (the agent is not a controller-runtime manager): watermark/lag/progress gauges,keys_applied_total,forced_resync_total{reason}(every known reason pre-seeded at 0 soincrease()catches the first resync), drift gauges, andtls_cert_expiry_timestamp_seconds{side,kind}read from the cert files on a refresh ticker./healthz(process liveness) and/readyz(past Connecting and not Failed; Degraded and Drained both count as ready — Degraded is backoff, not death, and a drained agent must keep serving its terminal status)./statuszso the controller can observe the terminal snapshot; exit codes distinguish clean shutdown, transient-loop exit, and permanent failure. A second SIGTERM/SIGINT breaks a wedged shutdown.Packaging
Both binaries ship in the existing single image (
ENTRYPOINTstays/manager; the Deployment setscommand: ["/mirror-agent"]).make buildproducesbin/mirror-agentalongsidebin/manager.Tests
Unit tests cover config translation/validation (quantities, endpoint normalization, reconcile translation, repeatable exclude-prefix), TLS wiring (per-handshake reload, bundle precedence, material-without-tls rejection), auth file handling,
/statusz//readyzsemantics per phase, the metrics collector (including conditional series absence), and cert-expiry gauges with generated certs; a live test drives the realrun()wiring against two embedded etcds.A kind e2e (
TestMirrorAgent) runs the shipped image in-cluster: two size-1EtcdClusters plus a mirror-agent pod; 10 keys written to the source are readable from the target in ~330ms, a delete propagates in ~330ms, and/statusz+/metricsare asserted to advance (keysAppliedTotal≥ writes, watermark movement, phase Syncing). The suite'sgo testtimeout is raised to 30m to keep worst-case waits inside the binary timeout.Verified with
make test,make verify,make build, and the kind e2e run.PR series — operability fixes, TLS & EtcdMirror
Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.
events.k8s.ioRBAC so operator Events are actually recordedmembers[]/leaderIDfrom one health snapshot (consistent leader)altNames.ipAddressesinto certificatesvalidityDuration(365d,100d12h) as documentedDegradedcondition (was empty status)--max-concurrent-reconciles)podTemplate.specscheduling fields: topologySpread, resources, priorityClass, schedulerNameconfig/surfacespec.tls.{peer,client}surfaces (breaking alpha API)TLSReadycondition + TLS lifecycle EventsPeerCANotSharedEtcdMirrorCRD: one-way cross-cluster replication API (types + CEL)pkg/mirroragentreplication engine (fenced checkpoint-in-target)mirror-agentbinary: config/TLS-reload/statusz/metrics + kind e2e/metricsendpointEtcdBackupCR → object storage (S3/GCS)🟢 ready · ⚪ draft