test: failing reproductions of sync bugs (evidence, do not merge)

[gmaclennan]

Important

Not for merging. This branch exists as evidence: each [BUG …] test below reproduces a bug in the current sync code and fails by design, so CI on this PR is red on purpose. The proposed fix is the sync refactor in #1304, which references these bugs by ID.

The sync subsystem has a history of intermittent, hard-to-diagnose bugs around sync completion, progress reporting, and device counts. To make those concrete, this branch adds black-box reproductions (plus a controllable transport for timing-dependent scenarios) against the current implementation. The bugs below stem from a code review by Claude Fable 5.0, and some are hard/impossible to reproduce reliably with failing tests.

The bugs are grouped by theme:

• A* = leave/block/close lifecycle
• B = completion detection
• C = capability/permissions
• D = connection lifecycle
• E = progress and device-count reporting
• F = state-derivation math
• G = findings made later, during the refactor.
• P0.x = test-coverage gaps.

1. Bugs with failing reproductions

ID	Bug	Repro
A1	SyncApi has no close(): its creator-core listeners, autostop timer, pending-discovery-key timers, and server websockets are never torn down when a project closes	test-e2e/sync-lifecycle.js
A3/A4	Closing a project mid-sync leaves sync listeners live on closed cores (and on real-latency links the in-flight core.update({wait:true}) rejects unhandled)	test-e2e/sync-lifecycle.js
P0.13	After a full project.close(), sync's peer-remove listeners remain on the creator core (the observable signal of A1/A3)	test-e2e/sync-lifecycle.js
P0.12	leaveProject() racing project.close() rejects with Cannot await idle after closing (partially demonstrates A2 — see below)	test-e2e/sync-lifecycle.js
B3	The internal initial-sync wait used by the invite flow never resolves for a peer whose presync namespaces are blocked — it only ever times out	test-e2e/sync-completion.js
C1	After blocking a peer, any unrelated auth write re-caches that peer's capability without preserving the auth workaround — auth replication stops and the blocked peer can never learn it was blocked	test-e2e/sync-capability.js
D1	Overlapping reconnect from the same device: the stale connection's teardown deletes the live connection's entries, so a connected device disappears from sync state and completion misbehaves	test-e2e/sync-reconnect.js
F1	mutatingAddPeerState uses === where = was intended, so a peer's aggregated per-namespace status is order-dependent — feeding wrong isSyncEnabled reporting and the gate that enables data sync	test/sync/sync-state-edge-cases.js
G1	A device holding a stale role record for a removed device starts replicating config immediately on connect (and data after presync) before syncing and indexing the auth records that would reveal the removal — new data leaks to a removed device. The capability re-read also races record indexing (syncThrottleMs: 0 + a realistic batch of auth records makes this deterministic)	test-e2e/sync-capability.js

2. Confirmed in review, no deterministic repro

These were verified by reading the code paths but are either RTT-bounded races that don't reproduce on loopback, or design-level inconsistencies with no single failing assertion. This section is their record.

• A2 — leaveProject()'s kClearData closes writer cores and purges/compacts other members' data while sync is still active with connected peers (a blanket sync-stop would be wrong: the leave flow needs auth to keep replicating to propagate the LEFT role). The exposed window is CoreSyncState listeners running derivation on closed cores during the post-leave auth-resync. P0.12 above demonstrates one symptom of this window.
• B1 — waitForSync('initial') can resolve before late-discovered cores replicate: per-core peer state is seeded only for cores that existed when the peer connected, and late-added cores never back-fill known peers. The isSynced guard is satisfied by sibling cores, so addProject can return before auth/config is actually pulled. RTT-bounded, so it passes on loopback; the ported test documents the structural gap in comments.
• B2 — getSyncStatus decides 'synced' from the namespace-global localState.want (a union across all peers) rather than per-peer counts, so enabling data sync with peer X can be held up by peer Y's in-flight transfer. Conservative in direction, but it consumes the same status-merge output as F1, so it is not purely cosmetic.
• C3 — capability reads (getRole) are unsequenced: two in-flight reads can resolve out of order and cache a stale capability. Self-heals on the next throttled state event; low individual harm, but it compounds the C1 path.
• D2 — the peer-disconnect handler's early-return (when no controller is found for the connection) skips disconnectPeer, leaving that peer's per-core state frozen forever.
• E1 — reported device state mixes the throttled state snapshot with a live capability read, so a single emitted state can be internally inconsistent.
• E2 — a group's isSyncEnabled can over-report true when one namespace in the group has no peer state yet (the aggregation skips it instead of counting it against the group).
• E3 — NO_ROLE/LEFT peers (whose auth capability is 'allowed') appear in remoteDeviceSyncState, so Object.keys(...).length is not a reliable count of actively-syncing devices — the source of the recurring device-count over/under-reporting.
• E4 — role transitions interact badly with the capability cache: a block can flip auth off via the C1 path, and a LEFT device can stay counted as syncing. Structurally the same root cause as C1/E3, listed separately because the user-visible symptom (wrong device counts during role changes) was reported independently.

3. Already fixed on main (kept as passing regression coverage)

• A0 — leaveProject() rejects with Sync timeout even though the local leave already succeeded: the trailing propagation wait rejects on timeout instead of being best-effort. Confirmed in production; fixed on main. The repro here ([A0 — fixed on main] in test-e2e/sync-lifecycle.js) passes and stays as regression coverage.
• F2 — scalar want(index) disagrees with have() and wantWord() at the contiguousLength boundary (> vs >=). No runtime impact (the scalar forms had no callers); fixed on main, covered by a passing unit test here.

4. Withdrawn after a product decision

• C2 — "a blocked peer's config/data/blobIndex cores are still added (and downloaded) locally — the gate checks for NO_ROLE only." Originally judged a bug (fetching a blocked device's data from third parties looked unintended). This is the desired behavior: blocking revokes what a device may receive (everything except auth, which carries its block record), not the project's claim on the data it contributed before the block — that data remains project data and must keep propagating, e.g. to a new device or archive that never met the blocked device.

5. Housekeeping (no behavior, no test)

• D3 — dead #downloadingRanges field in peer-sync-controller.js, declared but never read or written.

Note

CI runs npm test, which fail-fasts at the first failing suite — so the red check shows the first repro (F1, in the unit tests) rather than all of them. Run the command below to see every repro fail locally.

Running

node --test test-e2e/sync-lifecycle.js test-e2e/sync-completion.js \
  test-e2e/sync-capability.js test-e2e/sync-reconnect.js \
  test/sync/sync-state-edge-cases.js

test-e2e/controllable-wire.js (added here) drives two real projects over a pausable/latency transport via the kProjectReplicate seam; the syncThrottleMs option is threaded through manager → project → sync so tests can remove the 200ms state throttle where determinism matters.