Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
d86ffe0
Initial native CDC commit
tclinkenbeard-oai May 23, 2026
158333e
Add native CDC commit proxy routing
tclinkenbeard-oai May 23, 2026
9ce0954
Add native CDC stream lifecycle metadata operations and tests
tclinkenbeard-oai May 23, 2026
a28711d
Fix correctness regressions
tclinkenbeard-oai May 24, 2026
46e41a0
Add native CDC acknowledgement watermarks
tclinkenbeard-oai May 24, 2026
0bc5799
Add native CDC proxy consume and acknowledge backend
tclinkenbeard-oai May 24, 2026
7203fe5
Recruit and publish CDC proxy endpoints through recovery routing
tclinkenbeard-oai May 24, 2026
9f52b0c
Persist and publish CDC stream ownership routing
tclinkenbeard-oai May 25, 2026
9b76882
Add non-recovery CDC proxy failover and durable stream reassignment
tclinkenbeard-oai May 25, 2026
230af4d
Preserve native CDC streams across recovery and proxy failover
tclinkenbeard-oai May 25, 2026
92752c5
Add buffered native CDC consumption and safe shared-tag acknowledgements
tclinkenbeard-oai May 25, 2026
4d48b1c
Add client-facing native CDC operations with proxy failover retries
tclinkenbeard-oai May 25, 2026
e61abe0
Finalize CDC retention state when streams are removed
tclinkenbeard-oai May 25, 2026
364ec1e
Bound buffered mutation memory in CDC proxies
tclinkenbeard-oai May 25, 2026
a8599a4
Avoid waiting forever during concurrent CDC stream removal
tclinkenbeard-oai May 25, 2026
1808150
Bound CDC tags and persist stream ID allocation
tclinkenbeard-oai May 25, 2026
7acc707
Fix CDC proxy buffered byte release accounting
tclinkenbeard-oai May 25, 2026
dfb79ba
Resume CDC buffers from the retained minimum version
tclinkenbeard-oai May 25, 2026
f98c7f5
Refresh CDC buffer cursors after acknowledgements
tclinkenbeard-oai May 25, 2026
7ada95e
Anchor CDC retention at stream registration commit versions
tclinkenbeard-oai May 25, 2026
d3c942d
Register the CDC shared-tag simulator test
tclinkenbeard-oai May 25, 2026
eb58b1a
Fix Native CDC recovery progress across log generations
tclinkenbeard-oai May 26, 2026
7a9e54e
Contain Native CDC stream expiry without terminating its proxy
tclinkenbeard-oai May 26, 2026
f26afd0
Fix Native CDC workload consumption synchronization
tclinkenbeard-oai May 26, 2026
e0d06a4
Fix Native CDC recovery handling and stabilize simulator coverage
tclinkenbeard-oai May 27, 2026
9b4b2f5
Simplify serialize methods
tclinkenbeard-oai May 27, 2026
fc075ff
Move native CDC acknowledgements to storage-backed system keys
tclinkenbeard-oai May 27, 2026
7964d4f
Fix formatting
tclinkenbeard-oai May 27, 2026
74fb2b0
Prevent CDC stream removal from popping shared unread data
tclinkenbeard-oai May 27, 2026
eab9670
Fail CDC consume and acknowledge requests after stream removal
tclinkenbeard-oai May 27, 2026
d3fdfad
Update file_identifiers
tclinkenbeard-oai May 27, 2026
1fce475
Enable unit tests in simulation
tclinkenbeard-oai May 27, 2026
30ac0af
Use std::unordered_map for NativeCdcIdentifierAllocator::tagStreamCounts
tclinkenbeard-oai May 27, 2026
1810d8a
Add TODO comments for future CDC load balancing work
tclinkenbeard-oai May 27, 2026
9509829
Add ENABLE_NATIVE_CDC knob
tclinkenbeard-oai May 27, 2026
270b3d7
Fix CDC lifecycle recovery, retired-tag cleanup, and expiry handling
tclinkenbeard-oai May 27, 2026
c75f037
Allow native CDC consumption by registered stream name
tclinkenbeard-oai May 27, 2026
0323a68
Create native CDC cursors from registered stream names
tclinkenbeard-oai May 27, 2026
2e386e3
Clean up retired CDC tag pop state after confirmed drain
tclinkenbeard-oai May 27, 2026
807c252
Reject future native CDC acknowledgements to prevent premature log po…
tclinkenbeard-oai May 27, 2026
76ea730
Initial design/cdc.md draft
tclinkenbeard-oai May 27, 2026
e676f99
Bound CDC proxy buffering with a shared proxy-wide memory budget
tclinkenbeard-oai May 27, 2026
26dc480
s/first/current
tclinkenbeard-oai May 27, 2026
0a09b89
Remove redundant co_return statements
tclinkenbeard-oai May 27, 2026
6855c7c
Use mermaid for diagram
tclinkenbeard-oai May 27, 2026
ca11683
Share CDC tag readers across streams and retain log consumers during …
tclinkenbeard-oai May 28, 2026
e4d8604
Add fdbserver_cdcproxy_test target
tclinkenbeard-oai May 28, 2026
65e7d25
Add end-to-end native CDC simulation coverage with overlapping stream…
tclinkenbeard-oai May 28, 2026
3ff5ff6
Remove hard-coded configuration for NativeCdcEndToEnd.toml
tclinkenbeard-oai May 28, 2026
0761ac6
Fix native CDC failover assignment repair and acknowledged log poppin…
tclinkenbeard-oai May 28, 2026
db90538
Fix native CDC recovery and replication across failure topologies
tclinkenbeard-oai May 28, 2026
2823578
Add code probes for native CDC
tclinkenbeard-oai May 28, 2026
1dd8b80
Fix native CDC proxy reassignment wakeups and initialize streams befo…
tclinkenbeard-oai May 28, 2026
93065de
Fix native CDC CI linkage and allow draining streams after disablement
tclinkenbeard-oai May 28, 2026
185500c
Fix native CDC replay from satellite TLogs and add targeted coverage
tclinkenbeard-oai May 28, 2026
1bb10ee
Refine native CDC API boundaries and consumer coordination
tclinkenbeard-oai May 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
564 changes: 564 additions & 0 deletions design/cdc.md

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions fdbclient/ClientKnobs.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,8 @@ void ClientKnobs::initialize(Randomize randomize, IsSimulated isSimulated) {
init( CHANGE_FEED_CACHE_FLUSH_BYTES, 10e6 ); if( randomize && BUGGIFY ) CHANGE_FEED_CACHE_FLUSH_BYTES = deterministicRandom()->randomInt64(1, 1e6);
init( CHANGE_FEED_CACHE_EXPIRE_TIME, 60.0 ); if( randomize && BUGGIFY ) CHANGE_FEED_CACHE_EXPIRE_TIME = 1.0;
init( CHANGE_FEED_CACHE_LIMIT_BYTES, 500000 ); if( randomize && BUGGIFY ) CHANGE_FEED_CACHE_LIMIT_BYTES = 50000;
init( ENABLE_NATIVE_CDC, false ); if( randomize && isSimulated && BUGGIFY ) ENABLE_NATIVE_CDC = true;
init( NATIVE_CDC_TAG_COUNT, 256 ); if( randomize && BUGGIFY ) NATIVE_CDC_TAG_COUNT = 2;

init( MAX_BATCH_SIZE, 1000 ); if( randomize && BUGGIFY ) MAX_BATCH_SIZE = 1;
init( GRV_BATCH_TIMEOUT, 0.005 ); if( randomize && BUGGIFY ) GRV_BATCH_TIMEOUT = 0.1;
Expand Down
Loading
Loading