test: add ZooKeeper integration runners

[weicao]

Summary

• add local ZooKeeper source/target services for dt-tests
• add one-way and cycle ZK integration runners
• make ZK test endpoints configurable through the existing .env substitution path
• assert live shadow metadata, tombstones, markers, data convergence, and anti-loop version stability
• require the live data + live shadow precondition before testing delete/tombstone convergence

Candidate pin

• stacked base: PR feat: add ZooKeeper dual-active registry sync connector #504 branch feat/zk-connector-interface
• base commit tested: de88a916afd6c4e6b84cfc3fb359133069bc9cfc
• initial patch commit: 547164793135e4b150a3f623ab4e2fa86d38ef62
• current patch head: e2c2f293b9ab7381fe6f55a6a9a2c411afc411a3

Verification

Local gates on the current head:

• git diff --check: PASS
• cargo fmt --all --check: PASS
• cargo test --package dt-tests --test integration_test --no-run: PASS
• ordinary clippy: PASS with six pre-existing warnings outside the changed file
• independent 8-class design-contract review by @vicky: no blocker

Manual GitHub workflows on the exact current head (the stacked base does not auto-trigger pull-request workflows):

• CI run 29128577141: 2/2 jobs success
• Integration Tests run 29128577120: 33/33 jobs success

Focused IDC2 ZooKeeper runtime on the exact current head:

• one-way cdc_basic_test: N=1 PASS, 1 passed / 0 failed, 3.62s
• cycle cdc_cycle_basic_test: N=1 PASS, 1 passed / 0 failed, 19.19s
• source and target cleanup: /app, /__ape_dts_shadow, and /__ape_dts_marker have no ape-dts residue

The original head produced one focused one-way FAIL because the runner created and immediately deleted svc-gamma before the extractor recorded its live state. The current head fixes that Layer 3 test-ordering defect with bounded live-precondition and tombstone-convergence checks; the focused rerun then passed.

Boundary

This PR remains stacked on unmerged PR #504. The evidence above is exact-head static/compile, one manual CI workflow, one manual Integration Tests workflow, and focused IDC runtime N=1+1. It is sufficient for code review but is not a full-acceptance or release-ready claim. If PR #504 merges to a different final SHA, this branch must be re-pinned and affected gates rerun before retargeting to main.

[weicao]

Manual stacked-branch verification completed for patch commit 547164793135e4b150a3f623ab4e2fa86d38ef62 on base pin de88a916afd6c4e6b84cfc3fb359133069bc9cfc:

• CI run 29113409552: completed / success, 2 of 2 jobs successful (Tests, Lint & Static Checks), non-success jobs = 0.
• Integration Tests run 29113409612: completed / success, 33 of 33 jobs successful (discovery + 32 integration jobs), non-success jobs = 0.

Why manual runs were used: this PR is intentionally stacked on feat/zk-connector-interface; the repository pull-request workflows only auto-trigger for base main.

Boundary: this is one manual CI workflow and one manual integration workflow for the pinned commit. It is not fresh IDC ZooKeeper runtime evidence (runtime N=0) and is not a release-ready claim. The PR remains draft. If PR #504 merges to a different final SHA, re-pin and rerun affected gates before retargeting this PR to main.

[weicao]

Focused IDC runtime follow-up for exact patch commit e2c2f293b9ab7381fe6f55a6a9a2c411afc411a3:

• One-way cdc_basic_test: PASS, 1 passed / 0 failed, 3.62s, Saiyin, 2026-07-11 11:49 CST.
• Two-way cdc_cycle_basic_test: PASS, 1 passed / 0 failed, 19.19s, Saiyin, 2026-07-11 11:51 CST.
• Cleanup: source 192.168.10.41:32347 and target 192.168.10.41:31188 both reported zero ape_dts-related paths under /app, /__ape_dts_shadow, and /__ape_dts_marker.

The previous one-way failure on 547164793135e4b150a3f623ab4e2fa86d38ef62 was classified as a Layer 3 test-semantics failure: the runner created and deleted svc-gamma before the extractor had observed the live node. Commit e2c2f293 adds bounded convergence for live data + live shadow before delete, then bounded convergence for target absence + complete tombstone, and aborts the task on error paths. The fixed one-way sample closes that specific first blocker.

Additional exact-SHA gates:

• CI run 29128577141: completed / success, 2 of 2 jobs successful.
• Integration Tests run 29128577120: completed / success, 33 of 33 jobs successful.
• Independent review gates: git diff --check, cargo fmt --all --check, and cargo test --package dt-tests --test integration_test --no-run passed.

Boundary: runtime evidence is N=1 one-way + N=1 two-way focused coverage. It is not full-suite repeated acceptance or release-ready evidence. This PR remains stacked on PR #504; if PR #504 merges at a different final SHA, re-pin and rerun affected gates before retargeting.

[weicao]

Exact-head focused ZooKeeper runtime verification completed for e2c2f293b9ab7381fe6f55a6a9a2c411afc411a3 on IDC2 via Saygin:

• One-way command:
  cargo test --package dt-tests --test integration_test zk_to_zk::cdc_tests::test::cdc_basic_test -- --nocapture
  • result: PASS, 1 passed / 0 failed
  • duration: 3.62s
• Cycle command:
  cargo test --package dt-tests --test integration_test zk_to_zk::cdc_tests::test::cdc_cycle_basic_test -- --nocapture
  • result: PASS, 1 passed / 0 failed
  • duration: 19.19s
• Cleanup: source 192.168.10.41:32347 and target 192.168.10.41:31188 both have zero ape-dts residue under /app, /__ape_dts_shadow, and /__ape_dts_marker.

Failure/fix evidence: the previous head 547164793135e4b150a3f623ab4e2fa86d38ef62 produced one focused one-way FAIL because svc-gamma was created and deleted before the extractor had recorded its live state. The current head waits, within a bounded window, for target live data plus live shadow before deleting the source node, then waits for target absence plus a complete tombstone. The same one-way case passed after that test-ordering fix; the cycle case also passed.

Exact-head supporting gates:

• CI 29128577141: 2/2 jobs success
• Integration Tests 29128577120: 33/33 jobs success
• independent design-contract review: no blocker

Boundary: this is focused runtime N=1+1 plus exact-head CI/general integration evidence. It is not full acceptance or release readiness. PR #533 is stacked on unmerged PR #504; after #504 merges, re-pin #533 to the final merge SHA and rerun affected gates before retargeting to main.

[weicao]

Focused ZooKeeper repeat round 2 completed on the unchanged exact head e2c2f293b9ab7381fe6f55a6a9a2c411afc411a3:

• one-way zk_to_zk::cdc_tests::test::cdc_basic_test: PASS, 1 passed / 0 failed, 3.68s
• cycle zk_to_zk::cdc_tests::test::cdc_cycle_basic_test: PASS, 1 passed / 0 failed, 19.22s
• cleanup: source 192.168.10.41:32347 and target 192.168.10.41:31188 both have zero ape-dts paths under /app, /__ape_dts_shadow, and /__ape_dts_marker

Cumulative current-head focused runtime:

• one-way: N=2, PASS=2, FAIL=0
• cycle: N=2, PASS=2, FAIL=0
• both rounds closed with zero residue on the three recorded prefixes

Boundary: this is focused N=2+2 runtime evidence for the exact stacked-PR head. It is not full acceptance or release readiness. PR #533 still depends on PR #504; after #504 merges, re-pin to the final main SHA and rerun affected gates before retargeting.