[CORE-16776] cluster_link: test SR-sync against a Confluent source by mnajda-redpanda · Pull Request #31049 · redpanda-data/redpanda

mnajda-redpanda · 2026-07-08T12:03:04Z

Adds end-to-end ducktape coverage for the shadow-link Schema Registry
"API mode" sync against a Confluent Platform source registry (backed by
an Apache Kafka cluster), alongside the existing Redpanda-to-Redpanda
coverage, and fixes the one product gap it surfaced. The suite is
restructured into a shared mixin with a thin leaf per source vendor, so
the same tests run against either source through a _make_source_client
hook.

Standing up the Confluent source revealed that soft-delete propagation
did not converge cross-vendor: the reconciler took an imported version's
deleted-state from the per-version source response body, which a
standard source (e.g. Confluent) omits by default, so a soft-deleted
version imported active and the full-sync diff re-imported it every
cycle, forever. The fix derives the deleted-state from the caller's
authoritative active-vs-deleted listing partition instead.

Release Notes

none

Copilot

Pull request overview

Adds cross-vendor end-to-end ducktape coverage for shadow-link Schema Registry “API mode” sync using a Confluent Platform Schema Registry + Apache Kafka as the source, and fixes soft-delete propagation so it converges even when the source vendor omits deleted from per-version responses.

Changes:

Add a Confluent Schema Registry ducktape service and docker image dependency to run it in CI.
Refactor the SR sync E2E test into a shared base with two leaf variants (Redpanda-source and Confluent-source), and make secondary broker count configurable to fit node budgets.
Fix SR soft-delete replication by threading an authoritative “soft-deleted” partition (work_set.soft_deleted) into the reconciler import path; update unit tests and the mirroring task accordingly.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/type-checking/type-check-strictness.json	Adds the new Confluent SR service file to the type-check strictness mapping.
tests/rptest/tests/cluster_linking_test_base.py	Makes secondary cluster broker count configurable via `secondary_num_brokers`.
tests/rptest/tests/cluster_linking_schema_registry_sync_test.py	Splits SR sync E2E into a base + per-vendor leaf tests; adds Confluent-source topology and hooks.
tests/rptest/services/confluent_schema_registry.py	Introduces a ducktape `Service` wrapper to run a single-node Confluent Schema Registry.
tests/docker/ducktape-deps/confluent-schema-registry	Downloads/pins Confluent Platform Community tarball for Schema Registry.
tests/docker/Dockerfile	Adds a build stage and copies `/opt/confluent` into the final ducktape image.
src/v/cluster_link/schema_registry_sync/tests/reconciler_test.cc	Updates/extends unit tests to cover caller-authoritative soft-delete behavior.
src/v/cluster_link/schema_registry_sync/reconciler.h	Extends `work_set` with `soft_deleted` and tracks it in the reconciler.
src/v/cluster_link/schema_registry_sync/reconciler.cc	Forces imported deleted-state from `_soft_deleted` rather than trusting per-version bodies.
src/v/cluster_link/schema_registry_sync/mirroring_task.cc	Populates `work.soft_deleted` from the source listing partition during full sync.

mnajda-redpanda · 2026-07-08T13:16:26Z

force-pushed: fixed copilot comments

dotnwat · 2026-07-08T18:59:05Z

+            // reconciler imports these soft-deleted regardless of the
+            // per-version source body, which a standard source (e.g. Confluent)
+            // does not populate with a `deleted` flag by default.
+            work.soft_deleted.insert(node);


It is unclear to me what this part of the commit message means

The reconciler took an imported version's soft-delete state from the
per-version source response body. A standard source (e.g. Confluent)
omits the deleted flag from that body by default, so a soft-deleted
source version imported active and the full-sync diff re-imported it
every cycle, never converging. (Confluent can be coaxed into emitting
the flag with the Confluent-Accept-Unknown-Properties: true request
header.)

It sounds like you are saying that the fix in this commit is not needed if something else behaves differently (e.g. Confluent-Accept-Unknown-Properties: true is set). Is that accurate?

Reworded this bit. The current fix works regardless of whether we get the deleted property, it derives the state from the listing and never reads the body flag. If we do get it, this will work as a fallback (that's @pgellert comment below, I'll try to handle that in a next force-push)

EDIT: will do that in a followup PR

dotnwat · 2026-07-08T19:05:02Z

+            f"listeners=http://0.0.0.0:{self.port}\n"
+            f"kafkastore.bootstrap.servers={self._bootstrap_servers()}\n"
+            "kafkastore.topic=_schemas\n"
+            "schema.compatibility.level=none\n"


is compat level something we need to eventually surface into the test matrix?

We could, but it is just a lower-layer default of the compatibility level returned in GET /config, so we are already replicating it.

dotnwat · 2026-07-08T19:09:11Z

+    def _source_sr_url(self) -> str:
+        # Derive from the seeding client so the link source_url and the client
+        # can never point at different registries.
+        return self._make_source_client().base_uri()


Replacements below with self._source_sr_url() had been using self.source_cluster_service.schema_reg(limit=1), and now they use base_uri(). Are those equivalent?

Yes, they are

dotnwat · 2026-07-08T19:12:28Z

+        # Skip SchemaRegistrySyncE2EBase.__init__, which wires a Redpanda
+        # secondary SR; the source here is Apache Kafka + Confluent SR, so go
+        # straight to the shadow-link base with only the destination SR set.


this is confusing. are we inheriting from SchemaRegistrySyncE2EBase then just to get some code reuse? If so, then maybe we need to break out functionality into a MixIn instead?

I'm wondering how hard it would be to hook up these tests to be parametrizable on the source cluster like we do it for AK vs Redpanda source clusters:

redpanda/tests/rptest/tests/cluster_linking_e2e_test.py

Lines 1180 to 1181 in 7299281

SecondaryClusterSpec(

ServiceType.KAFKA, kafka_version="3.8.0", kafka_quorum="COMBINED_KRAFT"

with just a simple selector for Redpanda SR or Confluent SR as the source. And then we can just parametrize all these tests on the source.

Moved to mixins, if it's fine with you I'd leave it like this.

dotnwat · 2026-07-08T19:14:10Z

+    # Two-broker Kafka source so the topology uses exactly the inherited
+    # num_nodes budget (see class docstring).
+    secondary_num_brokers = 2


i don't understand this. is it configuring something in a super class?

Yes, it was. Removed and passed this as part of SecondaryClusterArgs. This controls the size of source cluster nodes.

pgellert · 2026-07-09T11:33:30Z

+    schema.deleted = _soft_deleted.contains(n) ? ppsr::is_deleted::yes
+                                               : ppsr::is_deleted::no;


I'm not opposed to turning this into a follow-up task, but I think if we do get the deleted field on the response, then we should prefer that over the _soft_deleted set, because:

The deleted field on the response is more up to date (came from a later API call than the list versions earlier)

The field will be present in the majority of the cases

It is not derived from a set difference between two API calls that may race with a low probability

It would force us to represent the optional-presence of the deleted field in the rest client's response object, which makes this behaviour more explicit to any other future call sites as well.

Lets tackle this in a followup

pgellert · 2026-07-09T12:46:25Z

+        # Skip SchemaRegistrySyncE2EBase.__init__, which wires a Redpanda
+        # secondary SR; the source here is Apache Kafka + Confluent SR, so go
+        # straight to the shadow-link base with only the destination SR set.


I'm wondering how hard it would be to hook up these tests to be parametrizable on the source cluster like we do it for AK vs Redpanda source clusters:

redpanda/tests/rptest/tests/cluster_linking_e2e_test.py

Lines 1180 to 1181 in 7299281

SecondaryClusterSpec(

ServiceType.KAFKA, kafka_version="3.8.0", kafka_quorum="COMBINED_KRAFT"

with just a simple selector for Redpanda SR or Confluent SR as the source. And then we can just parametrize all these tests on the source.

pgellert · 2026-07-09T12:53:25Z

+            f"listeners=http://0.0.0.0:{self.port}\n"
+            f"kafkastore.bootstrap.servers={self._bootstrap_servers()}\n"
+            "kafkastore.topic=_schemas\n"
+            "schema.compatibility.level=none\n"


We could, but it is just a lower-layer default of the compatibility level returned in GET /config, so we are already replicating it.

mnajda-redpanda · 2026-07-09T15:34:58Z

force-pushed: fixed comments from @dotnwat

move source broker count from secondary_num_brokers onto SecondaryClusterArgs(num_brokers=...)
restructure the SR-sync suite into a shared mixin with per-source leaves

mnajda-redpanda · 2026-07-09T17:26:42Z

force-pushed: to resolve merge conflicts

vbotbuildovich · 2026-07-09T18:49:55Z

Retry command for Build#86930

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/cluster_linking_schema_registry_sync_test.py::ConfluentSchemaRegistrySyncE2ETest.test_schema_registry_api_sync_e2e

vbotbuildovich · 2026-07-09T19:24:03Z

CI test results

test results on build#86930

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	ShadowLinkTopicFailoverTests	test_link_failover	{"source_cluster_spec": {"cluster_type": "redpanda"}, "storage_mode": "tiered", "with_failures": false}	integration	https://buildkite.com/redpanda/redpanda/builds/86930#019f47f6-04df-4710-9f2c-de3450b4457d	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0002, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkTopicFailoverTests&test_method=test_link_failover
FAIL	ConfluentSchemaRegistrySyncE2ETest	test_schema_registry_api_sync_e2e	null	integration	https://buildkite.com/redpanda/redpanda/builds/86930#019f47f6-04e1-4c49-9171-fe92c2b64619	0/11	The test was found to be new, and no failures are allowed	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ConfluentSchemaRegistrySyncE2ETest&test_method=test_schema_registry_api_sync_e2e
FAIL	ConfluentSchemaRegistrySyncE2ETest	test_schema_registry_api_sync_e2e	null	integration	https://buildkite.com/redpanda/redpanda/builds/86930#019f47f8-fd72-4e90-b582-332ac8e9e17c	0/11	The test was found to be new, and no failures are allowed	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ConfluentSchemaRegistrySyncE2ETest&test_method=test_schema_registry_api_sync_e2e

test results on build#86955

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	NodeWiseRecoveryTest	test_node_wise_recovery	{"dead_node_count": 2}	integration	https://buildkite.com/redpanda/redpanda/builds/86955#019f4bbf-6d9a-4273-90b9-74deaa87bef2	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0246, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_node_wise_recovery

mnajda-redpanda · 2026-07-10T10:36:32Z

force-pushed:

confluent leaf now runs the full 9-test suite (added compatibility,
hard-delete, context-remap delegators).
relax the compat-counter zero-check on the Confluent leaf, with a
comment on the :.__GLOBAL: cross-vendor gap.

second force-push: to resolve merge conflict

The reconciler took an imported version's soft-delete state from the per-version source response body. That body field cannot be relied on across SR vendors: a standard source (e.g. Confluent) omits the `deleted` flag by default, so a soft-deleted source version imported active and the full-sync diff re-imported it every cycle, never converging. Rather than depend on that field, derive the deleted-state from the active-vs-deleted listing partition, which every vendor reports reliably. Thread it from the caller through work_set.soft_deleted into import_body, which sets the imported deleted-state from it and no longer reads the body flag at all. Converges in both directions. reconciler_test gains a regression test where the source body reports a version active but the caller declares it soft-deleted.

Download Confluent Platform Community (7.7.1) into the test-node image and add a single-node ConfluentSchemaRegistryService that runs the registry against any bootstrap_provider (an Apache Kafka or Redpanda cluster). Used as a cross-vendor source in the shadow-link SR tests.

Split the SR-sync e2e test into a non-collected mixin holding the shared logic plus per-source leaves: the existing Redpanda-source test (same test-ids) and a new Confluent-source leaf backed by an Apache Kafka cluster. Source registry access goes through _make_source_client / _source_sr_url hooks so the same validation surface runs against either vendor. Move the source broker count onto SecondaryClusterArgs so a leaf sizes its source where it configures it, instead of a class attribute the base reads back. The Confluent leaf holds its Kafka source to a single broker, fitting a 5-node budget (3 dest + 1 kafka + 1 SR); the Confluent SR needs a node of its own, and a one-broker source is enough with the SR's _schemas at RF=1.

github-actions Bot added the area/redpanda label Jul 8, 2026

mnajda-redpanda self-assigned this Jul 8, 2026

mnajda-redpanda force-pushed the mnajda/confluent-source-sync branch from 0808316 to 150f550 Compare July 8, 2026 12:53

mnajda-redpanda marked this pull request as ready for review July 8, 2026 12:57

mnajda-redpanda requested a review from a team as a code owner July 8, 2026 12:57

mnajda-redpanda requested review from bartoszpiekny-redpanda, cjayani, Copilot, dotnwat, nguyen-andrew and pgellert and removed request for a team July 8, 2026 12:57

Copilot started reviewing on behalf of mnajda-redpanda July 8, 2026 12:57 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread tests/rptest/services/confluent_schema_registry.py

Comment thread tests/rptest/tests/cluster_linking_schema_registry_sync_test.py Outdated

Comment thread tests/rptest/tests/cluster_linking_schema_registry_sync_test.py Outdated

mnajda-redpanda force-pushed the mnajda/confluent-source-sync branch from 150f550 to e6b4d21 Compare July 8, 2026 13:16

dotnwat removed the request for review from cjayani July 8, 2026 18:51

dotnwat reviewed Jul 8, 2026

View reviewed changes

pgellert reviewed Jul 9, 2026

View reviewed changes

mnajda-redpanda force-pushed the mnajda/confluent-source-sync branch from e6b4d21 to 4bf7cc2 Compare July 9, 2026 15:34

mnajda-redpanda force-pushed the mnajda/confluent-source-sync branch from 4bf7cc2 to 7c7622f Compare July 9, 2026 17:26

mnajda-redpanda force-pushed the mnajda/confluent-source-sync branch from 7c7622f to d22b228 Compare July 10, 2026 10:35

mnajda-redpanda added 3 commits July 10, 2026 12:39

mnajda-redpanda force-pushed the mnajda/confluent-source-sync branch from d22b228 to 589e7dd Compare July 10, 2026 11:03

pgellert approved these changes Jul 10, 2026

View reviewed changes

mnajda-redpanda merged commit b6d43ac into redpanda-data:dev Jul 10, 2026
23 checks passed

mnajda-redpanda deleted the mnajda/confluent-source-sync branch July 10, 2026 13:16

	SecondaryClusterSpec(
	ServiceType.KAFKA, kafka_version="3.8.0", kafka_quorum="COMBINED_KRAFT"

		schema.deleted = _soft_deleted.contains(n) ? ppsr::is_deleted::yes
		: ppsr::is_deleted::no;

Uh oh!

Conversation

mnajda-redpanda commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mnajda-redpanda commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mnajda-redpanda Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mnajda-redpanda commented Jul 9, 2026

Uh oh!

mnajda-redpanda commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vbotbuildovich commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Retry command for Build#86930

Uh oh!

vbotbuildovich commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI test results

Uh oh!

mnajda-redpanda commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mnajda-redpanda commented Jul 8, 2026 •

edited

Loading

mnajda-redpanda Jul 9, 2026 •

edited

Loading

mnajda-redpanda commented Jul 9, 2026 •

edited

Loading

vbotbuildovich commented Jul 9, 2026 •

edited

Loading

vbotbuildovich commented Jul 9, 2026 •

edited

Loading

mnajda-redpanda commented Jul 10, 2026 •

edited

Loading