Skip to content

Commit ac60243

Browse files
committed
cluster_link/replication: retry deferred prefix truncation on commit
When a source partition is prefix-trimmed, the shadow partition has to truncate to the same start offset. maybe_synchronize_start_offset() defers that while the shadow hasn't replicated up to the source start offset yet and (before this commit) relied on the next fetch_next() to retry. It's possible that after deferring the prefix-trim, we replicate up to the source HWM so there isn't a next fetch_next(), and so our start offset never proceeds. This commit retries the synchronization when a batch commits, which is exactly when the shadow high watermark advances past the offset the truncation was waiting on, so it no longer depends on new source data arriving. This attempts to fix ShadowLinkingReplicationTests.test_auto_prefix_trimming which is quite flaky.
1 parent d013af2 commit ac60243

2 files changed

Lines changed: 21 additions & 0 deletions

File tree

src/v/cluster_link/replication/partition_replicator.cc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,9 @@ ss::future<bool> partition_replicator::handle_replication_result(
121121
// source to sink
122122
// We only intend to backoff on consecutive failures.
123123
_backoff_policy.reset();
124+
// The shadow high watermark just advanced, so a prefix truncation that
125+
// was deferred waiting for the shadow to catch up can now proceed.
126+
co_await maybe_synchronize_start_offset();
124127
co_return true;
125128
} catch (...) {
126129
auto eptr = std::current_exception();

src/v/cluster_link/replication/tests/partition_replicator_tests.cc

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -408,4 +408,22 @@ TEST_F_CORO(PartitionReplicatorFixture, TestNoStartOffsetOvershootOnResume) {
408408
5s, [&] { return _sink->start_offset() == kafka::offset(50); });
409409
}
410410

411+
// A shadow partition that catches up to a prefix-trimmed source in a single
412+
// batch must still sync its start offset. When the batch is fetched the shadow
413+
// HWM is still behind the trim point, so the truncation is deferred; and once
414+
// the batch is replicated no further source data arrives to drive another
415+
// fetch. The truncation must therefore be retried when the batch commits.
416+
TEST_F_CORO(PartitionReplicatorFixture, TestTrimSyncAfterSingleCatchUpFetch) {
417+
// Source is prefix-trimmed to 5 but still has a moving tail (lso > 5), so
418+
// truncation stays deferred until the shadow catches up to the trim point.
419+
_source->set_reported_start_offset(kafka::offset(5));
420+
421+
// A single batch [0, 9] catches the shadow up past the trim point. After it
422+
// commits the source is idle, so the commit itself must retry the sync.
423+
co_await push_data();
424+
425+
RPTEST_REQUIRE_EVENTUALLY_CORO(
426+
5s, [&] { return _sink->start_offset() == kafka::offset(5); });
427+
}
428+
411429
} // namespace cluster_link::replication

0 commit comments

Comments
 (0)