Skip to content

Commit ff5256d

Browse files
committed
cluster_link: reset SR-sync state on losing _schemas leadership
The mirroring task instance is created once per link and reused across leadership changes: losing leadership only stops the runner, so its in-memory status, destination inventory, and last-full-sync timestamp survived. On an A->B->A handoff the returning leader would report a prior tenure's stale counters and could skip its first full sync while _last_full_sync was still recent, deferring catch-up by up to the full-sync interval. Reset the derived state in stop() (safe there: task::stop() has closed the runner gate, so no run_impl is in flight). _config and the pending _config_changed flag are preserved, so a config change queued while stopped still takes effect. Every leadership acquisition now re-derives from the durable destination store, matching a genuinely fresh instance. Extend the leadership-change e2e to bounce _schemas/0 leadership back to the original broker and assert the reused instance also starts fresh: a later start_time, zeroed totals, and a full sync that imports nothing.
1 parent 3f94488 commit ff5256d

3 files changed

Lines changed: 70 additions & 3 deletions

File tree

src/v/cluster_link/schema_registry_sync/mirroring_task.cc

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,8 +97,22 @@ void mirroring_task::update_config(const model::metadata& link_metadata) {
9797
_config_changed = true;
9898
}
9999

100+
void mirroring_task::reset_sync_state() {
101+
_status = model::schema_registry_sync_status{};
102+
_destination_inventory = inventory{};
103+
_reconcile_stats = reconcile_stats{};
104+
_last_full_sync.reset();
105+
}
106+
100107
ss::future<cl_result<void>> mirroring_task::stop() noexcept {
101108
auto res = co_await task::stop();
109+
// task::stop() closed the runner's gate, so no run_impl is in flight and it
110+
// is safe to reset the state directly (unlike update_config, which races a
111+
// running fiber and defers via _config_changed). Reset so a later leader
112+
// starts fresh: this instance may regain _schemas/0 leadership (A->B->A)
113+
// and would otherwise report a prior tenure's stale counters/inventory and
114+
// skip its first full sync on a still-recent _last_full_sync.
115+
reset_sync_state();
102116
// The run loop has stopped, so no fiber is using the reader; release its
103117
// HTTP transport. as_future guards the noexcept contract.
104118
if (_reader) {
@@ -425,9 +439,10 @@ ss::future<task::state_transition> mirroring_task::full_source_sync(
425439

426440
ss::future<task::state_transition>
427441
mirroring_task::run_impl(ss::abort_source& as) {
428-
// Stamp the cumulative summary's start time once per task instance (the
429-
// proto documents totals_since_task_start.start_time as the task start).
430-
// A fresh instance after a leadership change re-stamps it on its first run.
442+
// Stamp the cumulative summary's start time once per leadership acquisition
443+
// (the proto documents totals_since_task_start.start_time as the task
444+
// start). stop() clears the status on losing leadership, so the next leader
445+
// -- a new instance or this same one regaining it -- re-stamps it here.
431446
if (!_status.totals_since_task_start.start_time.has_value()) {
432447
_status.totals_since_task_start.start_time = ::model::timestamp::now();
433448
}

src/v/cluster_link/schema_registry_sync/mirroring_task.h

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,14 @@ class mirroring_task : public task {
8787
/// new source URL, auth, or TLS setting takes effect on the next run.
8888
ss::future<> reset_reader();
8989

90+
/// Clears the in-memory sync state (status counters, destination inventory,
91+
/// last-full-sync timestamp) on losing leadership, so the next leader -- a
92+
/// new instance or this same one regaining leadership -- re-derives
93+
/// everything from the durable destination store instead of a prior
94+
/// tenure's view. `_config`/`_config_changed` are preserved: config is
95+
/// authoritative and a change queued while stopped must still take effect.
96+
void reset_sync_state();
97+
9098
/// Whether a periodic full scan is due (first run, or the full-sync
9199
/// interval has elapsed). A config change additionally forces one via
92100
/// `_config_changed`, consumed in `run_impl`.

tests/rptest/tests/cluster_linking_schema_registry_sync_test.py

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -792,3 +792,47 @@ def re_derived() -> bool:
792792
backoff_sec=1,
793793
err_msg="new leader did not count the post-bounce import",
794794
)
795+
796+
# Now bounce leadership back to the original broker (A -> B -> A). Its
797+
# task instance ran before, was stopped on losing leadership, and is now
798+
# reused -- not a fresh object. stop() clears the in-memory sync state,
799+
# so the returning leader must look just as fresh as B did: a later
800+
# start_time, zeroed cumulative counters, and an immediate full sync (a
801+
# stale _last_full_sync would otherwise defer it up to the full-sync
802+
# interval). Without the reset it would report its pre-B totals instead.
803+
expected_subjects += 1 # post-bounce-value
804+
expected_versions += 1
805+
before_return_start = (
806+
self._admin_sr_status().totals_since_task_start.start_time.ToNanoseconds()
807+
)
808+
admin.partition_transfer_leadership(
809+
namespace="kafka", topic="_schemas", partition=0, target_id=leader
810+
)
811+
admin.await_stable_leader(
812+
topic="_schemas",
813+
partition=0,
814+
namespace="kafka",
815+
timeout_s=60,
816+
check=lambda node_id: node_id == leader,
817+
)
818+
819+
def returning_leader_fresh() -> bool:
820+
sr = self._admin_sr_status()
821+
totals = sr.totals_since_task_start
822+
return (
823+
sr.HasField("last_full_sync")
824+
and totals.start_time.ToNanoseconds() > before_return_start
825+
and totals.subject_versions_changed == 0
826+
and totals.errors == 0
827+
and sr.last_full_sync.subject_versions_changed == 0
828+
and sr.last_full_sync.errors == 0
829+
and sr.inventory.selected_source_subjects == expected_subjects
830+
and sr.inventory.destination_subject_versions == expected_versions
831+
)
832+
833+
wait_until(
834+
returning_leader_fresh,
835+
timeout_sec=90,
836+
backoff_sec=1,
837+
err_msg="reused instance did not reset state on regaining leadership",
838+
)

0 commit comments

Comments
 (0)