feat: add ZooKeeper dual-active registry sync connector by weicao · Pull Request #504 · apecloud/ape-dts

weicao · 2026-06-06T13:22:50Z

Summary

Add ZooKeeper CDC extractor (scan-based) and sinker with full dual-active bidirectional sync support
Implement anti-loop mechanism using DataMarker (source_id in marker znode) and skip-if-unchanged data comparison
Support LWW (Last Write Wins) conflict resolution with source_id tie-break for deterministic ordering
Add ZkFilter (path-prefix whitelist/blacklist + ephemeral filtering) and ZkRouter (path-prefix remapping)

Key Components

Component	File	Lines
ZkExtractor	`dt-connector/src/extractor/zk/zk_extractor.rs`	~350
ZkSinker	`dt-connector/src/sinker/zk/zk_sinker.rs`	~290
ZkFilter	`dt-common/src/zk_filter.rs`	~45
ZkRouter	`dt-connector/src/zk_router.rs`	~45
ZK types	`dt-common/src/meta/zk/`	~45

Anti-Loop Design

In dual-active mode, two tasks run simultaneously (node1→node2 and node2→node1). Without anti-loop:

node1→node2 syncs data → version bump on node2
node2→node1 detects "change" → syncs back → version bump on node1
Infinite loop

Solution: Sinker compares data bytes before writing. If target znode already has identical data, the write is skipped — no version bump, no loop trigger.

Test Plan

Single-direction sync (node1→node2): data and content verified
Reverse sync (node2→node1): data and content verified
Dual-active simultaneous: both tasks running, sinked_records stabilizes (no infinite loop)
Data consistency: both ZK instances have identical znodes after sync
Docker test environment: dual ZK 3.9 containers (ports 2181/2182)
Compilation: cargo check --workspace and cargo build --release pass

🤖 Generated with Claude Code

Related Issue

Fixes #523

weicao · 2026-06-11T13:03:51Z

IDC2 ZooKeeper runtime acceptance for this PR head is complete enough to move out of draft.

Pinned candidate:

Head SHA: b1ab772338b5b1c5b05d80dafdb71131f8f49446
IDC2 source ZK: 192.168.10.41:32347
IDC2 target ZK: 192.168.10.41:31188

Runtime evidence:

zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: N=3 PASS
zk_to_zk::cdc_tests::test::cdc_basic_test: N=2 PASS
Total accepted focused runtime: N=5, PASS=5, FAIL=0, SKIP=0
Cleanup after runs: /app, /__ape_dts_shadow, and /__ape_dts_marker absent on both IDC2 endpoints.

Boundary:

This is focused IDC2 runtime evidence for PR review readiness, not full release readiness.
During test expansion, one dt-tests config issue was found and fixed locally: do_paths=* was treated as a literal prefix by the ZK filter, so the test config now uses do_paths=/app,/config. The failed pre-fix run is classified as test probe/config, not product behavior.

weicao · 2026-06-11T13:45:54Z

Follow-up for the new head after rebasing/merging current main into this PR branch.

Pinned candidate:

Head SHA: c7b403dec2a641e1a5d27c6e47a94be2ec3de131

Why the branch changed:

Ready-for-review CI initially failed on the synthetic merge ref because current main refactored dt-connector/src/rdb_router.rs and RdbTopicRouterInner::from_config matched only RouterConfig::Rdb.
This PR adds the missing neutral/default handling for non-RDB configs there, matching the existing RDB route behavior.

Verification on the new head:

cargo fmt --all --check: PASS
cargo test --no-run --message-format json-render-diagnostics --package dt-tests --test integration_test: PASS
With the local ZK dt-tests scaffold applied for runtime only: cargo test --package dt-tests --test integration_test --no-run: PASS
IDC2 focused runtime on this new head: zk_to_zk::cdc_tests::test::cdc_basic_test: N=1 PASS
IDC2 focused runtime on this new head: zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: N=1 PASS
Post-run cleanup proof: /app, /__ape_dts_shadow, and /__ape_dts_marker absent on both IDC2 ZooKeeper endpoints after manual narrow cleanup.

Boundary:

The earlier N=5 focused runtime evidence belongs to old head b1ab772338b5b1c5b05d80dafdb71131f8f49446.
Because the PR head changed, this comment records fresh focused evidence for c7b403dec2a641e1a5d27c6e47a94be2ec3de131: N=2 PASS so far, not full release readiness.
GitHub CI has restarted for the new head; lint/tests and many integration jobs are already green, with some integration jobs still pending at the time of this comment.

weicao · 2026-06-11T14:05:06Z

Follow-up: GitHub CI is now green for head c7b403dec2a641e1a5d27c6e47a94be2ec3de131.

Completed checks:

Lint & Static Checks: PASS
Tests: PASS
Integration Tests: all discovered integration jobs PASS, including mysql_to_mysql and pg_to_pg.

Review boundary remains the same as the previous comment: current-head focused IDC2 ZK runtime evidence is N=2 PASS so far; the older N=5 focused runtime evidence belongs to previous head b1ab772338b5b1c5b05d80dafdb71131f8f49446 and is historical support only.

weicao · 2026-06-11T15:24:57Z

Additional focused IDC2 runtime expansion for current head c7b403dec2a641e1a5d27c6e47a94be2ec3de131:\n\n- zk_to_zk::cdc_tests::test::cdc_basic_test: one additional PASS at 2026-06-11 23:23 Asia/Shanghai, finished in 11.84s.\n- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: one additional PASS at 2026-06-11 23:24 Asia/Shanghai, finished in 21.80s.\n- Current-head focused runtime total is now N=4, PASS=4, FAIL=0, SKIP=0: one-way N=2 PASS, cycle N=2 PASS.\n- Post-run cleanup proof after manual narrow cleanup: /app, /__ape_dts_shadow, and /__ape_dts_marker are absent on both IDC2 ZooKeeper endpoints.\n\nBoundary: this remains focused IDC2 runtime evidence for review readiness, not full release readiness. The earlier N=5 focused runtime evidence belongs to old head b1ab772338b5b1c5b05d80dafdb71131f8f49446 and is historical support only.

weicao · 2026-06-11T16:28:59Z

Additional focused IDC2 runtime expansion for current head c7b403dec2a641e1a5d27c6e47a94be2ec3de131:\n\n- zk_to_zk::cdc_tests::test::cdc_basic_test: one additional PASS at 2026-06-12 00:27 Asia/Shanghai, finished in 14.40s.\n- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: one additional PASS at 2026-06-12 00:28 Asia/Shanghai, finished in 20.01s.\n- Current-head focused runtime total is now N=6, PASS=6, FAIL=0, SKIP=0: one-way N=3 PASS, cycle N=3 PASS.\n- Post-run cleanup proof after manual narrow cleanup: /app, /__ape_dts_shadow, and /__ape_dts_marker are absent on both IDC2 ZooKeeper endpoints.\n\nBoundary: this remains focused IDC2 runtime evidence for review readiness, not full release readiness. The earlier N=5 focused runtime evidence belongs to old head b1ab772338b5b1c5b05d80dafdb71131f8f49446 and is historical support only.

weicao · 2026-06-11T16:59:59Z

Current-head focused IDC2 runtime expansion for PR #504 head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d):

zk_to_zk::cdc_tests::test::cdc_basic_test: PASS at 2026-06-12 00:58 Asia/Shanghai, finished in 12.21s
zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: PASS at 2026-06-12 00:58 Asia/Shanghai, finished in 21.74s

Current-head focused total is now N=8, PASS=8, FAIL=0, SKIP=0:

one-way case: N=4, PASS=4
cycle case: N=4, PASS=4

Narrow cleanup proof after this run:

source 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
target 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: this is focused IDC2 runtime evidence for the current PR head and review readiness. It is not a full release-ready claim. The older b1ab772 N=5 evidence remains historical support only.

weicao · 2026-06-11T17:32:01Z

Current-head focused IDC2 runtime expansion for PR #504 head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d):

zk_to_zk::cdc_tests::test::cdc_basic_test: PASS at 2026-06-12 01:31 Asia/Shanghai, finished in 11.68s
zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: PASS at 2026-06-12 01:31 Asia/Shanghai, finished in 20.06s

Current-head focused total is now N=10, PASS=10, FAIL=0, SKIP=0:

one-way case: N=5, PASS=5
cycle case: N=5, PASS=5

Narrow cleanup proof after this run:

source 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
target 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: this is focused IDC2 runtime evidence for the current PR head and review readiness. It is not a full release-ready claim. The older b1ab772 N=5 evidence remains historical support only.

weicao · 2026-06-11T18:04:07Z

Current-head focused IDC2 runtime expansion for PR #504 head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d):

zk_to_zk::cdc_tests::test::cdc_basic_test: PASS at 2026-06-12 02:03 Asia/Shanghai, finished in 11.62s
zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: PASS at 2026-06-12 02:03 Asia/Shanghai, finished in 20.18s

Current-head focused total is now N=12, PASS=12, FAIL=0, SKIP=0:

one-way case: N=6, PASS=6
cycle case: N=6, PASS=6

Narrow cleanup proof after this run:

source 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
target 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: this is focused IDC2 runtime evidence for the current PR head and review readiness. It is not a full release-ready claim. The older b1ab772 N=5 evidence remains historical support only.

weicao · 2026-06-11T18:36:54Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 02:35 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.75s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 20.01s

Current-head focused runtime total after this pair:

Total: N=14, PASS=14, FAIL=0, SKIP=0
One-way case: N=7 PASS
Cycle case: N=7 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T20:07:05Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 04:06 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.55s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 21.16s

Current-head focused runtime total after this pair:

Total: N=16, PASS=16, FAIL=0, SKIP=0
One-way case: N=8 PASS
Cycle case: N=8 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T20:39:16Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 04:38 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.56s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 19.99s

Current-head focused runtime total after this pair:

Total: N=18, PASS=18, FAIL=0, SKIP=0
One-way case: N=9 PASS
Cycle case: N=9 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T21:11:01Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 05:10 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.74s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 19.78s

Current-head focused runtime total after this pair:

Total: N=20, PASS=20, FAIL=0, SKIP=0
One-way case: N=10 PASS
Cycle case: N=10 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T21:44:05Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 05:43 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.95s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 20.35s

Current-head focused runtime total after this pair:

Total: N=22, PASS=22, FAIL=0, SKIP=0
One-way case: N=11 PASS
Cycle case: N=11 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Current PR route state observed before this run: PR remains open/non-draft at c7b403d; all visible GitHub checks were green; GitHub still reports review required / blocked, with no reviewer/assignee/review request visible.

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T22:16:49Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 06:16 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 12.06s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 20.44s

Current-head focused runtime total after this pair:

Total: N=24, PASS=24, FAIL=0, SKIP=0
One-way case: N=12 PASS
Cycle case: N=12 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Current PR route state observed before this run: PR remains open/non-draft at c7b403d; all visible GitHub checks were green; GitHub still reports review required / blocked, with no reviewer/assignee/review request visible.

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T22:48:47Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 06:48 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.99s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 20.14s

Current-head focused runtime total after this pair:

Total: N=26, PASS=26, FAIL=0, SKIP=0
One-way case: N=13 PASS
Cycle case: N=13 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Current PR route state observed before this run: PR remains open/non-draft at c7b403d; all visible GitHub checks were green; GitHub still reports review required / blocked, with no reviewer/assignee/review request visible.

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T23:21:54Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 07:21 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 12.08s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 20.20s

Current-head focused runtime total after this pair:

Total: N=28, PASS=28, FAIL=0, SKIP=0
One-way case: N=14 PASS
Cycle case: N=14 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Current PR route state observed before this run: PR remains open/non-draft at c7b403d; all visible GitHub checks are green; GitHub still reports review required / blocked, with no reviewer/assignee/review request visible. No Slock response was visible after the 06:51 route request to Musk3.

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-11T23:53:46Z

Current-head focused IDC2 runtime update for PR #504.

Scope: focused ZK dt-tests runtime on IDC2 endpoints, using the local dt-tests scaffold against PR head c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d). This is review-readiness evidence for the ZK connector path, not a full release-ready claim.

New run pair at 2026-06-12 07:53 Asia/Shanghai:

cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.79s
cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 19.93s

Current-head focused runtime total after this pair:

Total: N=30, PASS=30, FAIL=0, SKIP=0
One-way case: N=15 PASS
Cycle case: N=15 PASS

Cleanup proof after the run pair:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Current PR route state observed before this run: PR remains open/non-draft at c7b403d; all visible GitHub checks are green; GitHub still reports review required / blocked, with no reviewer/assignee/review request visible. No Slock response was visible after the 06:51 route request to Musk3.

Boundary: old-head evidence on b1ab772 remains historical support only; the current acceptance evidence above is pinned to c7b403d.

weicao · 2026-06-12T00:07:11Z

Current-head IDC2 ZK focused runtime update for c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d):

2026-06-12 08:05 CST: zk_to_zk::cdc_tests::test::cdc_basic_test PASS, 1 passed; 0 failed; 0 ignored; 374 filtered out, finished in 14.30s.
2026-06-12 08:06 CST: zk_to_zk::cdc_tests::test::cdc_cycle_basic_test PASS, 1 passed; 0 failed; 0 ignored; 374 filtered out, finished in 20.43s.
Current-head focused accepted runtime now: N=32, PASS=32, FAIL=0, SKIP=0; one-way N=16, cycle N=16.
Cleanup proof after the pair:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
Visible PR state at 2026-06-12 08:06 CST: open, non-draft, all visible checks pass, reviewDecision=REVIEW_REQUIRED, mergeStateStatus=BLOCKED, no assignee/review request/latest review returned by gh pr view.

Boundary: this is focused current-head review-readiness evidence for the ZK lane, not a full release-ready claim.

weicao · 2026-06-12T00:39:26Z

Current-head IDC2 ZK focused runtime update for c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d):

2026-06-12 08:38 CST: zk_to_zk::cdc_tests::test::cdc_basic_test PASS, 1 passed; 0 failed; 0 ignored; 374 filtered out, finished in 11.61s.
2026-06-12 08:38 CST: zk_to_zk::cdc_tests::test::cdc_cycle_basic_test PASS, 1 passed; 0 failed; 0 ignored; 374 filtered out, finished in 19.99s.
Current-head focused accepted runtime now: N=34, PASS=34, FAIL=0, SKIP=0; one-way N=17, cycle N=17.
Cleanup proof after the pair:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
Visible PR state at 2026-06-12 08:39 CST: open, non-draft, all visible checks pass, reviewDecision=REVIEW_REQUIRED, mergeStateStatus=BLOCKED, no assignee/review request/latest review returned by gh pr view.

Boundary: this is focused current-head review-readiness evidence for the ZK lane, not a full release-ready claim.

weicao · 2026-06-12T01:07:47Z

Current-head IDC2 ZK focused runtime update for c7b403dec2a641e1a5d27c6e47a94be2ec3de131 (c7b403d):

2026-06-12 09:06 CST: zk_to_zk::cdc_tests::test::cdc_basic_test PASS, 1 passed; 0 failed; 0 ignored; 374 filtered out, finished in 11.75s.
2026-06-12 09:06 CST: zk_to_zk::cdc_tests::test::cdc_cycle_basic_test PASS, 1 passed; 0 failed; 0 ignored; 374 filtered out, finished in 19.92s.
Current-head focused accepted runtime now: N=36, PASS=36, FAIL=0, SKIP=0; one-way N=18, cycle N=18.
Cleanup proof after the pair:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
Visible PR state at 2026-06-12 09:07 CST: open, non-draft, all visible checks pass, reviewDecision=REVIEW_REQUIRED, mergeStateStatus=BLOCKED, no assignee/review request/latest review returned by gh pr view.

Boundary: this is focused current-head review-readiness evidence for the ZK lane, not a full release-ready claim.

weicao · 2026-06-12T01:41:04Z

Current-head ZK focused follow-up for c7b403d after the 2026-06-12 09:38 keepalive.

Scope: the two targeted dt-tests that previously covered this PR's ZK regression surface, run against IDC2 endpoints zk_src_url=192.168.10.41:32347 and zk_dst_url=192.168.10.41:31188.

Results at 2026-06-12 09:39-09:40 Asia/Shanghai:

PASS zk_to_zk::cdc_tests::test::cdc_basic_test (cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture), finished in 12.08s.
PASS zk_to_zk::cdc_tests::test::cdc_cycle_basic_test (cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture), finished in 19.99s.
Current-head focused runtime total: N=38 targeted runs, PASS=38, FAIL=0, SKIP=0. Split: one-way N=19, cycle N=19.
Cleanup proof after the run:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

PR state checked at 2026-06-12 09:39 Asia/Shanghai: open, non-draft, head still c7b403d, visible required checks green, GitHub merge gate still REVIEW_REQUIRED / BLOCKED with no reviewer/assignee/review request visible from gh pr view.

Conclusion boundary: this is focused current-head runtime evidence for the ZK regression surface, not a full release claim. The remaining blocker appears to be human review / merge ownership, not CI or these targeted ZK tests.

weicao · 2026-06-12T02:13:35Z

Current-head ZK focused follow-up for c7b403d after the 2026-06-12 10:11 keepalive.

Scope: the two targeted dt-tests that cover this PR's ZK regression surface, run against IDC2 endpoints zk_src_url=192.168.10.41:32347 and zk_dst_url=192.168.10.41:31188.

Results at 2026-06-12 10:12-10:13 Asia/Shanghai:

PASS zk_to_zk::cdc_tests::test::cdc_basic_test (cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture), finished in 11.65s.
PASS zk_to_zk::cdc_tests::test::cdc_cycle_basic_test (cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture), finished in 19.84s.
Current-head focused runtime total: N=40 targeted runs, PASS=40, FAIL=0, SKIP=0. Split: one-way N=20, cycle N=20.
Cleanup proof after the run:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

PR state checked at 2026-06-12 10:11 Asia/Shanghai: open, non-draft, head still c7b403d, visible checks green, GitHub merge gate still REVIEW_REQUIRED / BLOCKED with no reviewer/assignee/review request visible from gh pr view.

Conclusion boundary: this is focused current-head runtime evidence for the ZK regression surface, not a full release claim. The remaining blocker appears to be human review / merge ownership, not CI or these targeted ZK tests.

weicao · 2026-06-12T02:46:08Z

Current-head ZK focused follow-up for c7b403d after the 2026-06-12 10:44 keepalive.

Scope: the two targeted dt-tests that cover this PR's ZK regression surface, run against IDC2 endpoints zk_src_url=192.168.10.41:32347 and zk_dst_url=192.168.10.41:31188.

Results at 2026-06-12 10:45 Asia/Shanghai:

PASS zk_to_zk::cdc_tests::test::cdc_basic_test (cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture), finished in 12.50s.
PASS zk_to_zk::cdc_tests::test::cdc_cycle_basic_test (cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture), finished in 19.94s.
Current-head focused runtime total: N=42 targeted runs, PASS=42, FAIL=0, SKIP=0. Split: one-way N=21, cycle N=21.
Cleanup proof after the run:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

PR state checked at 2026-06-12 10:44 Asia/Shanghai: open, non-draft, head still c7b403d, visible checks green, GitHub merge gate still REVIEW_REQUIRED / BLOCKED with no reviewer/assignee/review request visible from gh pr view.

Conclusion boundary: this is focused current-head runtime evidence for the ZK regression surface, not a full release claim. The remaining blocker appears to be human review / merge ownership, not CI or these targeted ZK tests.

weicao · 2026-06-12T03:19:50Z

Current-head PR #504 follow-up, 2026-06-12 11:20 Asia/Shanghai.

State checked before this run:

Head: c7b403d (c7b403d)
GitHub PR state: OPEN, non-draft
GitHub checks: all visible checks passing at the 11:17 check
GitHub gate still shown as REVIEW_REQUIRED / BLOCKED; no visible latestReviews, reviewRequests, or assignees in gh pr view

Additional focused IDC2 ZK runtime evidence on this same head:

env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 11.75s
env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 19.89s

Current-head focused total for c7b403d: N=44 PASS, FAIL=0, SKIP=0.

one-way cdc_basic_test: N=22 PASS
cycle cdc_cycle_basic_test: N=22 PASS

Post-run cleanup proof with Kazoo:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: this is focused current-head runtime regression evidence for the ZK connector surface plus CI state. It is not a full release claim. The visible blocker remains human review / merge ownership, not CI or the targeted ZK runtime pair.

weicao · 2026-06-12T03:52:32Z

Current-head PR #504 follow-up, 2026-06-12 11:52 Asia/Shanghai.

State checked before this run:

Head: c7b403d (c7b403d)
GitHub PR state: OPEN, non-draft
GitHub checks: all visible checks passing at the 11:50 check
GitHub gate still shown as REVIEW_REQUIRED / BLOCKED; no visible latestReviews, reviewRequests, or assignees in gh pr view

Additional focused IDC2 ZK runtime evidence on this same head:

env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 11.62s
env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 20.11s

Current-head focused total for c7b403d: N=46 PASS, FAIL=0, SKIP=0.

one-way cdc_basic_test: N=23 PASS
cycle cdc_cycle_basic_test: N=23 PASS

Post-run cleanup proof with Kazoo:

192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Boundary: this is focused current-head runtime regression evidence for the ZK connector surface plus CI state. It is not a full release claim. The visible blocker remains human review / merge ownership, not CI or the targeted ZK runtime pair.

weicao · 2026-06-12T04:25:57Z

Keepalive evidence update for PR #504 at c7b403d.

PR state refreshed at 2026-06-12 12:23 Asia/Shanghai: OPEN, non-draft, all visible checks passing, reviewDecision=REVIEW_REQUIRED, mergeStateStatus=BLOCKED, with no visible reviewer / review request / assignee in gh pr view.

Additional focused IDC2 runtime checks on current head:

one-way: env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 12.29s.
cycle: env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 20.05s.
Cleanup proof after the run: /app, /__ape_dts_shadow, and /__ape_dts_marker are absent on both 192.168.10.41:32347 and 192.168.10.41:31188.

Current-head focused ZK runtime total is now N=48 PASS / 0 fail / 0 skip since c7b403d became the PR head: one-way N=24, cycle N=24. Boundary: this is focused ZK connector runtime regression evidence plus visible CI status, not a full release-readiness claim. The visible gate remains human review / merge ownership.

weicao · 2026-06-12T04:58:12Z

Keepalive evidence update for PR #504 at c7b403d.

PR state refreshed at 2026-06-12 12:57 Asia/Shanghai: OPEN, non-draft, all visible checks passing, reviewDecision=REVIEW_REQUIRED, mergeStateStatus=BLOCKED, with no visible reviewer / review request / assignee in gh pr view.

Additional focused IDC2 runtime checks on current head:

one-way: env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 11.70s.
cycle: env zk_src_url=192.168.10.41:32347 zk_dst_url=192.168.10.41:31188 cargo test --package dt-tests --test integration_test -- zk_to_zk::cdc_tests::test::cdc_cycle_basic_test --nocapture -> PASS, 1 passed / 0 failed / 374 filtered out, finished in 19.71s.
Cleanup proof after the run: /app, /__ape_dts_shadow, and /__ape_dts_marker are absent on both 192.168.10.41:32347 and 192.168.10.41:31188.

Current-head focused ZK runtime total is now N=50 PASS / 0 fail / 0 skip since c7b403d became the PR head: one-way N=25, cycle N=25. Boundary: this is focused ZK connector runtime regression evidence plus visible CI status, not a full release-readiness claim. The visible gate remains human review / merge ownership.

weicao · 2026-06-12T05:33:03Z

PR #504 keepalive after reminder #c5e2fb98, current head c7b403d.

Refreshed GitHub state at 2026-06-12 13:29 Asia/Shanghai:

PR is OPEN and non-draft.
Visible checks are passing.
Merge gate is still human-review/ownership: mergeStateStatus=BLOCKED, reviewDecision=REVIEW_REQUIRED, no visible reviewer, review request, or assignee.

Focused ZK connector runtime evidence on IDC2 ZK endpoints, rerun sequentially after cleanup:

zk_to_zk::cdc_tests::test::cdc_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 11.94s.
zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: PASS, 1 passed / 0 failed / 374 filtered, finished in 19.50s.
Cleanup proof after the accepted pair:
- 192.168.10.41:32347: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent
- 192.168.10.41:31188: /app=absent, /__ape_dts_shadow=absent, /__ape_dts_marker=absent

Evidence boundary:

Current-head focused ZK connector total is now N=52 PASS / 0 accepted fail / 0 skip, split as one-way N=26 and cycle N=26.
Historical old-head b1ab772 evidence remains N=5 PASS and is not counted into the current-head total.
A concurrent run attempt at 13:29 had a shared test-path collision: the cycle test failed cleanup with delete_node /__ape_dts_shadow failed: node has not empty children while the one-way test was using the same IDC2 endpoints/paths. I classified that as Layer 2 test execution/harness interference, cleaned both endpoints, rejected that concurrent pair from the accepted count, and reran the one-way/cycle tests sequentially as listed above.
This is focused ZK connector regression evidence plus visible CI status. It is not a full release-readiness claim.

Remaining action: route human reviewer / merge owner for PR #504.

weicao · 2026-06-19T17:40:32Z

Technical route chosen after the 2026-06-20 #ape-dts follow-up.

PR #504 stays draft. We should not continue the old scan-polling extractor as the final production real-time sync route.

Chosen route: watch-triggered sync plus reconnect reconciliation.

Plain boundary:

Normal online path should use ZooKeeper watch events to reduce latency and avoid scanning the full tree every interval.
Reconnect/session-uncertain path must run bounded reconciliation to repair final state.
Product wording should be eventual-consistency ZooKeeper sync, not strict durable CDC.
ZooKeeper does not provide a database-binlog-style durable event history, so this route still cannot promise exact ordered replay for every change during disconnect.

Next implementation step:

keep this PR draft;
write/patch the extractor around watcher registration, event loop, reconnect reconciliation, and checkpoint semantics;
keep the current IDC2 N=490 result only as small-scope historical evidence for the old scan-based implementation, not as production-readiness evidence for the new design.

weicao · 2026-06-19T17:43:34Z

Design review addendum from the sinker side.

The watch-assisted route also needs a delete/tombstone contract; otherwise an older upsert can recreate data after a newer delete.

Required sink-side rule:

upsert and delete must use the same ordering comparison;
delete events must carry a comparable source order key, such as source mtime or source zxid; ZkStat::default() / mtime=0 is not valid for ordered delete;
after applying delete, do not immediately delete the shadow metadata;
instead write a tombstone shadow, for example {source_id, source_mtime/source_zxid, deleted:true};
older upserts must skip when the tombstone is newer;
newer upserts may recreate the business node and overwrite the tombstone shadow;
/__ape_dts_marker, /__ape_dts_shadow, and /__ape_dts_heartbeat must remain filtered from extractor/watch events.

This means loop prevention and conflict handling must be based on shadow source metadata for both present and absent business nodes, not only on marker paths.

weicao · 2026-06-19T20:36:36Z

Option B implementation plan after the 2026-06-20 design follow-up.

This is a plan, not implementation/runtime proof. PR #504 should stay draft until these contracts are in code and tested.

Product boundary

The target claim is eventual-consistency ZooKeeper sync.
This is not durable, ordered, lossless CDC.
ZooKeeper watches are triggers, not a history log.
Disconnect windows close by reconciliation to current state, not by replaying every missed event.

Code contracts before patching

Watch registration
- Register every configured watch_path with PersistentRecursive before the initial snapshot.
- Do not silently fall back to scan-primary mode if watch registration fails.
Watch consumption
- Every PersistentWatcher must be continuously polled through changed().
- Registering a watcher and not reading it is invalid; the crate warns this can grow memory.
Session uncertainty
- On Disconnected, Expired, watcher channel close, or source session loss, pause normal node-event classification.
- Reconnect/re-register watchers and run reconciliation before normal processing resumes.
Delete ordering
- Delete events must carry an explicit order key.
- Do not use ZkStat::default() / mtime=0 for deletes.
- Upserts can use source Stat.mtime; deletes need an explicit observed order, because ZooKeeper does not expose source-server delete mtime after the node is gone.
- If exact source-server delete time is required, ZooKeeper watches alone are insufficient.
Tombstone shadow
- Upsert writes shadow with deleted:false.
- Successful logical delete writes shadow tombstone with deleted:true; it does not delete shadow.
- Delete on a target path that is already absent is still a successful logical delete and must write tombstone.
- Older upsert skips when tombstone is newer; newer upsert may recreate and overwrite tombstone.
- NotEmpty delete remains a skip because the business node was not deleted; do not write tombstone for a skipped delete.
Internal paths
- Extractor/watch path handling must always filter /__ape_dts_marker, /__ape_dts_shadow, and /__ape_dts_heartbeat, even if task config forgets them.
Checkpoint wording
- Position::Zk.path_versions is a reconciliation index, not durable event history.
- If restart delete detection is kept, the path index remains necessary, so add explicit guardrails for max tracked paths / position bytes / reconciliation duration.

Patch order

Add explicit ZK source-order metadata and sinker tombstone unit tests.
Patch ZkSinker LWW/tombstone behavior.
Refactor ZkExtractor into watcher registration + event loop + shared reconciliation.
Add extractor tests for event mapping and session uncertainty.
Run runtime validation only through the approved test-agent environment; do not count old IDC2 N=490 as proof for this new implementation.

Local recovery note: full implementation plan is in Elizabeth's notes/zk-pr504-option-b-implementation-plan.md.

weicao · 2026-06-19T21:12:29Z

Pushed the first Option B code slice: metadata + sinker tombstone contract.

Head is now c843b6d804d98ddf9144f8f7c4a65e3aa80c729c.

What changed

ZkEntry now carries explicit source ordering metadata:
- source_order_millis
- source_zxid
- order_origin
Extractor-created upsert events now fill source order from source ZK stat.
Existing scan-detected delete events no longer use an incomparable mtime=0; they now carry an explicit observed order for reconciliation deletes.
ZkSinker LWW now compares the unified source order from shadow/tombstone metadata.
Successful logical delete writes a tombstone shadow with deleted:true instead of deleting shadow.
Existing old shadow JSON remains readable by falling back from source_order_millis to legacy source_mtime.

Tests run locally

cargo fmt --all --check: PASS
cargo test --package dt-connector sinker::zk::zk_sinker::tests -- --nocapture: PASS, 5/5
cargo check --package dt-connector --all-targets: PASS

Boundary

This is not runtime validation for Option B yet. PR remains draft.

Because the PR head changed from c7b403d to c843b6d8, the old IDC2 focused N=490 PASS remains historical evidence for the previous scan-based head only. Runtime evidence for this new head has not been collected yet.

weicao · 2026-06-19T21:16:53Z

Follow-up after @vicky's sinker-side contract review.

Head is now 755ab27362ad27482370907bdf8c6abfdabaff13.

Changes in 755ab273:

LWW ordering now compares (source_order_millis, source_zxid, source_id) instead of (source_order_millis, source_id).
Added zxid tie-break tests for same source + same millis:
- shadow/tombstone zxid newer => older upsert skips;
- entry zxid newer => newer upsert can pass older tombstone.
Tombstone shadow serialization now rejects ambiguous delete metadata:
- source_order_millis() must be positive;
- source_zxid must be positive.
Malformed shadow JSON now emits a warning before preserving the existing compatibility behavior of ignoring that shadow for LWW.

Local validation:

cargo fmt --all --check PASS
cargo test --package dt-connector sinker::zk::zk_sinker::tests -- --nocapture PASS (9 passed)
cargo check --package dt-connector --all-targets PASS
git diff --check PASS

Boundary: CI for 755ab273 is queued. This still does not include the watcher event-loop/reconnect reconciliation patch or new runtime validation.

weicao · 2026-06-19T21:27:00Z

Option B extractor slice pushed.

Head is now c4313d417ec22fc7a1a586c22a2393eda5d78b2f.

Changes in c4313d41:

Registers PersistentRecursive ZooKeeper watchers for all configured watch_paths before the initial snapshot/reconciliation.
Starts a continuous watcher consumer task for each watcher, so registered persistent watchers are drained instead of held idle.
Replaces the scan-primary sleep loop with a tokio::select! loop:
- watcher node events trigger scoped reconciliation for the affected path/subtree;
- periodic timer remains as a safety reconciliation path;
- session Disconnected marks the source uncertain;
- following SyncConnected / ConnectedReadOnly forces full reconciliation;
- terminal session states fail the extractor instead of continuing silently.
Refactors full scan and incremental rescan through a shared reconciliation function.
Adds always-on internal path filtering for /__ape_dts_marker, /__ape_dts_shadow, and /__ape_dts_heartbeat, even if task config whitelists them.
Adds focused unit tests for internal path filtering and scoped reconciliation path matching.

Local validation:

cargo fmt --all --check PASS
cargo test --package dt-common zk_filter::tests -- --nocapture PASS
cargo test --package dt-connector extractor::zk::zk_extractor::tests -- --nocapture PASS
cargo test --package dt-connector sinker::zk::zk_sinker::tests -- --nocapture PASS (9 passed)
cargo check --package dt-connector --all-targets PASS
git diff --check PASS

Boundary:

This is a watch-triggered reconciliation loop, not durable event replay.
Watch events are treated as triggers/hints; reconciliation emits current state.
No IDC runtime evidence exists yet for c4313d41.
GitHub CI for c4313d41 is queued at this comment time.

weicao · 2026-06-19T21:35:33Z

Follow-up for @vicky's reconciliation-read-error blocker.

Head is now 28d46de5aee9ca5d290a111d5990d1a0eeff8ccd.

Changes in 28d46de5:

scan_node_recursive now treats only ZkError::NoNode as "currently absent".
Non-NoNode get_data / list_children errors now abort the current reconciliation instead of logging and continuing with an incomplete scan.
Reconciliation now collects scan results/events first and emits entries only after the scan succeeds, so an aborted scan does not update path_versions and does not emit false delete tombstones.
Periodic reconciliation is skipped while session_uncertain=true; reconnect still forces full reconciliation through the watcher session path.
Internal path matching is exact-or-descendant instead of raw starts_with, so paths like /__ape_dts_shadow2 are not treated as internal.
Added focused tests for exact-or-descendant internal filtering and aborted-scan no-delete/no-position-replacement behavior.

Local validation:

cargo fmt --all --check PASS
cargo test --package dt-common zk_filter::tests -- --nocapture PASS
cargo test --package dt-connector extractor::zk::zk_extractor::tests -- --nocapture PASS (3 passed)
cargo test --package dt-connector sinker::zk::zk_sinker::tests -- --nocapture PASS (9 passed)
cargo check --package dt-connector --all-targets PASS
git diff --check PASS

Boundary: GitHub CI for 28d46de5 is queued. This still has no IDC runtime evidence yet.

weicao · 2026-06-19T21:44:30Z

Follow-up for @vicky's checkpoint boundary blocker.

Head is now c4c10d3061d178ba81d0bde87758c3da0a541ff0.

Changes:

Reconciliation-staged ZK entries now use Position::None instead of the final Position::Zk.
After all staged entries are enqueued, the extractor emits a trailing Heartbeat carrying the final reconciliation position.
This keeps checkpoint advancement behind the FIFO queue: if only a prefix of staged entries is drained/sunk, fetch_raw sees only Position::None; the final position is observed only when the trailing heartbeat is drained after the entries.
Added extractor coverage for staged ZK entry positions and pipeline coverage for fetch_raw advancing ZK position only on the trailing heartbeat.

Local validation:

cargo fmt --all --check PASS
cargo test --package dt-connector extractor::zk::zk_extractor::tests -- --nocapture PASS (4 passed)
cargo test --package dt-connector sinker::zk::zk_sinker::tests -- --nocapture PASS (9 passed)
cargo test --package dt-pipeline base_pipeline::tests -- --nocapture PASS (2 passed)
cargo check --package dt-connector --all-targets PASS
cargo check --package dt-pipeline --all-targets PASS
git diff --check PASS

Boundary:

GitHub CI for the new head is still running at comment time.
No IDC runtime evidence exists for this new head yet; old IDC2 N=490 belongs to the old c7b403d implementation.

…mplementation skeletons Adds the foundational type system for ZooKeeper dual-active registry sync: - Position::Zk with path_versions checkpoint, high_water_zxid, total_paths - DtData::Zk variant with ZkEntry (path, data, ZkStat, ephemeral, event_type, source_id) - ExtractorConfig::Zk / SinkerConfig::Zk with config loading from ini - ZkFilter (path prefix whitelist/blacklist) and ZkRouter (path prefix remapping) - ZkExtractor / ZkSinker skeleton structs with todo!() implementations - DataMarker ZK support (marker znode path matching and source_id refresh) - ConflictPolicyEnum::LastWriteWins for ZK conflict resolution - DbType::Zk and factory wiring in ExtractorUtil / SinkerUtil Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…Extractor, ZkRouter to ZkSinker Blocker fixes: - ZkExtractor: add extract_state, syncer, recovery fields (matching MongoCdcExtractor pattern); remove standalone monitor field - ZkSinker: add ZkRouter field for path prefix remapping Should-fix: - ZkFilter config loaded from [zk_filter] ini section instead of hardcoded; added zk_filter field to TaskConfig with load_zk_filter_config() - ZkRouter: changed from HashMap to Vec<(String, String)> sorted by prefix length descending for deterministic longest-prefix-match - ZkEntry.get_data_size(): include path length in size calculation Nit: - Removed redundant DbType::Zk arm in DataMarker::from_config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replaces the todo!() stub with complete sink implementation: - Lazy ZK client connection via zookeeper-client crate - Serial processing: route path → LWW conflict check → apply event - Event dispatch: Created (upsert), Updated (auto-create), Deleted (idempotent) - LWW conflict policy: compare target stat.mtime vs source mtime - ensure_parent_paths for recursive znode creation - DataMarker writing after each batch (JSON source_id in marker znode) - Ephemeral-to-persistent conversion via sync_ephemeral_as_persistent flag - Metrics tracking via base_sinker.update_serial_monitor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…node 1. LWW deterministic tie-break: read target marker znode once per batch to get existing source_id; when mtime equal, larger source_id wins; if target has no source_id (first sync), default apply 2. Delete NotEmpty handling: log_warn and skip instead of bail, avoids force-deleting parent nodes that have new children from peer 3. write_data_marker uses src_node (fixed identity) instead of data_origin_node (dynamic, refreshed per-event) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Full ZkExtractor implementation (~340 lines): - full_scan: recursive znode traversal with path/ephemeral filtering - incremental_rescan: fresh map scan + diff for delete detection, old_versions for version comparison to skip unchanged znodes - Event type correctly set to Created vs Updated based on old_versions - source_id from DataMarker.src_node (not random UUID) for anti-loop - Checkpoint recovery via Position::Zk (path_versions + high_water_zxid) - Heartbeat task writing source_id + position to /__ape_dts_heartbeat Cherry-picked from Elizabeth feat/zk-extractor-impl branch (d33512de). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- ZkExtractor: clone position before move to fix borrow checker error - base_struct_sinker, clickhouse_struct_sinker, starrocks_struct_sinker: add wildcard arm for ConflictPolicyEnum::LastWriteWins - base_pipeline: add DtData::Zk to SinkMethod::Raw dispatch All changes pass cargo check --workspace. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- docker-compose.zk.yml: two standalone ZK 3.9 instances (ports 2181, 2182) - zk_to_zk/cdc/basic_test/task_config.ini: ZK-to-ZK sync task config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Prevents infinite sync loop in dual-active mode by comparing data bytes before writing. If the target znode already has identical data, the write is skipped, avoiding version bumps that would trigger the peer extractor. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ZOO_MY_ID=2 caused config mismatch in standalone ZK 3.9 (only server.1 defined). Changed to ZOO_MY_ID=1 for both independent instances. Added topo1 node1↔node2 test configs with data_marker anti-loop. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace target-stat-based LWW with shadow metadata at /__ape_dts_shadow/<path> storing (source_id, source_mtime, version). This compares original source mtimes instead of sinker write times, enabling correct conflict resolution in dual-active mode. Also write/delete shadow alongside data operations and add /__ape_dts_shadow to ignore_paths in all test configs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Extractor now checks both version AND mzxid when determining if a path is already synced. Fixes missed sync after delete+recreate where version resets to 0 but mzxid advances. - Run cargo fmt to fix CI lint gate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The same-data early return skipped write_shadow, leaving shadow source_mtime stale. LWW decisions then used an outdated watermark. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

weicao marked this pull request as draft June 11, 2026 09:07

weicao marked this pull request as ready for review June 11, 2026 13:03

weicao marked this pull request as ready for review June 22, 2026 21:04

wei and others added 18 commits June 23, 2026 15:10

test(zk): add dual ZK docker-compose and basic test config

769981d

- docker-compose.zk.yml: two standalone ZK 3.9 instances (ports 2181, 2182) - zk_to_zk/cdc/basic_test/task_config.ini: ZK-to-ZK sync task config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(zk-sinker): update shadow metadata on skip-if-unchanged path

767d4bc

The same-data early return skipped write_shadow, leaving shadow source_mtime stale. LWW decisions then used an outdated watermark. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix: handle zk config in rdb topic router

0488c39

fix(zk): add ordered tombstone shadow handling

1834569

fix(zk): tighten tombstone lww ordering

7411bfb

feat(zk): add watch-triggered reconciliation loop

298e080

fix(zk): abort reconciliation on read uncertainty

a60f822

fix(zk): checkpoint reconciliation after staged entries

6b41e7c

weicao force-pushed the feat/zk-connector-interface branch from c4c10d3 to 6b41e7c Compare June 23, 2026 13:25

fix(ci): restore rust 1.85 compatibility

c1b4ea3

weicao mentioned this pull request Jul 1, 2026

ZooKeeper dual-active sync can miss watch-triggered reconciliation and delete tombstone safety #523

Open

Uh oh!

Conversation

weicao commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Components

Anti-Loop Design

Test Plan

Related Issue

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 11, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 12, 2026

Uh oh!

weicao commented Jun 19, 2026

Uh oh!

weicao commented Jun 19, 2026

Uh oh!

weicao commented Jun 19, 2026

Product boundary

Code contracts before patching

Patch order

Uh oh!

weicao commented Jun 19, 2026

What changed

Tests run locally

Boundary

Uh oh!

weicao commented Jun 6, 2026 •

edited

Loading