feat: add EtcdMirror CRD for one-way cross-cluster replication (types and CEL only)#406
feat: add EtcdMirror CRD for one-way cross-cluster replication (types and CEL only)#406xrl wants to merge 2 commits into
Conversation
Introduces the EtcdMirror CRD (schema only, no controller/agent) that will describe a continuous one-way key-range sync from a source etcd cluster to a target etcd cluster. Adds EtcdMirrorSpec/Status, the EtcdMirrorEndpoint/TLS/Auth/Sync/Checkpoint/Reconciliation sub-specs, phase and condition constants (including TargetThrottled and ReplicationLagExceeded), printer columns, and CEL XValidation rules for the destPrefix/noDestPrefix mutual exclusion, the endpointList/serviceRef oneOf, and the insecureSkipVerify/ insecureSkipVerifyAcknowledgeRisk companion-field requirement. Registers EtcdMirror/EtcdMirrorList in the scheme, regenerates zz_generated.deepcopy.go and the CRD manifest via controller-gen, and adds a realistic sample CR. CEL rules are covered by a table-driven envtest suite in etcdmirror_cel_test.go, following the tls_cel_test.go pattern, exercised against a real envtest apiserver. The design's Metrics field (reusing EtcdClusterSpec's MetricsSpec) is intentionally omitted: MetricsSpec does not exist on this branch yet and will land with pr/domain-metrics; it will be added in a later PR once that type is importable. No controller or agent logic in this PR -- types, generated code, and CRD schema only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Xavier Lange <xrlange@gmail.com>
Reshape the EtcdMirror API for the Design-3 engine (checkpoint stored in
the target etcd, stateless agent) per the 48-finding cross-cloud review.
Deletions:
- checkpoint.storageSpec (no PVC; checkpoint lives in the target at a
reserved fenced key)
- expectEmptyPrefix (superseded by initialSync.mode)
- noDestPrefix (redundant; one anchored rewrite formula
key' = target.prefix + destPrefix + TrimPrefix(key, source.prefix))
- syncInterval
Additions/renames:
- spec.mode: Sync|Drain; status.cutover{drainTargetRevision,
drainedRevision, verifiedTime, counts, leasedKeyCount}
- initialSync.mode: RequireEmpty|Overwrite|OverwriteAndPrune,
startRevision
- sync: requestTimeout, excludePrefixes, pageKeyLimit,
watchBufferBytes; batching flushes only at source-revision boundaries
- TLS: secretRef pointerized (nil = system trust roots), caBundleRef
- pod resources; configurable reserved checkpoint key
- watermark-derived lag/liveness semantics; frozen condition vocabulary
(TargetQuotaExhausted, ResyncLoopDetected, CutoverReady,
InvariantsHeld, LearnerEndpoint, Prefix/DirectionConflict) and Event
Reason constants as API contract
- status: source/target versions and cluster IDs, leaseBackedKeyCount,
always-on source/target key counts, initialSyncTotalKeyCount
- CEL: scheme-vs-TLS rules, prefix/rewrite immutability
- docs/etcdmirror.md (fidelity caveats, one-way contract) and
regenerated API reference
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Xavier Lange <xrlange@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xrl The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @xrl. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
1 participant
Adds the
EtcdMirrorCRD: a declarative, continuous one-way key-range replication link from a source etcd cluster to a target etcd cluster (types, generated code, CRD schema, CEL validation and docs only — the replication engine and controller follow in separate PRs; the engine library PR is next in the series).The API encodes the design's core decisions so they are contract, not implementation detail:
mod_revisioncompare on the checkpoint key, so two agents (or a pre-cutover straggler) can never interleave writes silently.spec.mode: Sync | Drainwith astatus.cutoverblock (drainTargetRevision,drainedRevision,verifiedTime, per-side key counts,leasedKeyCount). Cutover is gated on theCutoverReadycondition, which requires Drain mode and a reached drain target revision —InvariantsHeldalone never means "safe to cut over".initialSync.mode: RequireEmpty | Overwrite | OverwriteAndPrunereplaces a booleanexpectEmptyPrefix;OverwriteAndPruneis what makes reversal-by-recreate workable. Two-way replication is a non-goal by data model, not by roadmap: etcd revisions are cluster-local and carry no provenance channel, so bidirectional sync cannot be made loop-safe.key' = target.prefix + destPrefix + TrimPrefix(key, source.prefix).TargetQuotaExhausted,ResyncLoopDetected,CutoverReady,InvariantsHeld,LearnerEndpoint,PrefixConflict,DirectionConflict, …) and Event/Reason constants declared up front as API contract — later PRs add setters, never names. The condition comments carry the paging algebra (what pages immediately, what self-heals) so operators can alert on the API doc alone.ResyncLoopDetectedlivelock signature.Two placement choices worth calling out (and the reasoning, since both had a plausible alternative):
startRevisionsits underinitialSync, not at spec top level: its only valid pairing is withinitialSync.mode(it seeds a fidelity-preserving restore viaetcdutl snapshot restore --bump-revision --mark-compacted), and the CEL rule that enforces the pairing reads cleaner within one struct.resourcessits atspec.resourcesrather than inside a shared PodTemplate type: PodTemplate is shared with EtcdCluster, and coupling EtcdMirror's much smaller surface to it would inherit fields (persistence, member scheduling) that have no meaning for a stateless agent.Validation: table-driven CEL envtest suite (
etcdmirror_cel_test.go, following thetls_cel_test.gopattern) covering the endpoint oneOf, scheme-vs-TLS rules, conditional TLS secretRef, initialSync/startRevision pairing and prefix/rewrite immutability, against a real envtest apiserver.zz_generated.deepcopy.go, the CRD manifest anddocs/api-references/docs.mdare regenerated in the same commits; verified withmake testandmake verify.PR series — operability fixes, TLS & EtcdMirror
Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.
events.k8s.ioRBAC so operator Events are actually recordedmembers[]/leaderIDfrom one health snapshot (consistent leader)altNames.ipAddressesinto certificatesvalidityDuration(365d,100d12h) as documentedDegradedcondition (was empty status)--max-concurrent-reconciles)podTemplate.specscheduling fields: topologySpread, resources, priorityClass, schedulerNameconfig/surfacespec.tls.{peer,client}surfaces (breaking alpha API)TLSReadycondition + TLS lifecycle EventsPeerCANotSharedEtcdMirrorCRD: one-way cross-cluster replication API (types + CEL)pkg/mirroragentreplication engine (fenced checkpoint-in-target)mirror-agentbinary: config/TLS-reload/statusz/metrics + kind e2e/metricsendpointEtcdBackupCR → object storage (S3/GCS)🟢 ready · ⚪ draft