MINOR: Move client test utilities to test fixtures and resolve shadow jar conflicts by soarez · Pull Request #22201 · apache/kafka

soarez · 2026-05-04T01:48:39Z

This change moves TestSslUtils, TestUtils and accessory classes from the
test classpath in clients to test fixtures, allowing other modules to
use these classes for testing without having to include all of the test
classpath for clients.

For further details on the context and purpose of this change, check
these threads:
#20376 (comment)
#22193 (comment)

Reviewers: Chia-Ping Tsai chia7712@gmail.com

This change moves TestSslUtils, TestUtils and accessory classes from the test classpath in clients to test fixtures, allowing other modules to use these classes for testing without having to include all of the test classpath for clients. For further details on the context and purpose of this change, check these threads: apache#20376 (comment) apache#22193 (comment)

chia7712 · 2026-05-04T18:56:40Z

    shadowed
  }

+  // java-test-fixtures creates dependencies testImplementation -> testFixturesApi -> (main artifact)


This is definitely a magic workaround... I noticed there are several issues discussing this behavior

gradle/gradle#19946
gradle/gradle#11696

Anyway, good job finding this approach!

chia7712 · 2026-05-04T18:59:52Z


  apply plugin: 'java-library'
+
+  // Resolve conflict between testFixtures and shadow in :clients.


Could you add a comment referencing the "magic workaround" in the clients project?

chia7712 · 2026-05-05T02:10:14Z

@@ -1120,6 +1139,7 @@ project(':core') {
    implementation libs.re2j

    testImplementation project(':clients').sourceSets.test.output


It would be great if we could remove all references to project(':clients').sourceSets.test.output. We could move the classes used across modules to the test-fixtures folder. WDYT?

All such references were removed and disallowed - adding them now triggers a build error.

chia7712

LGTM

chia7712 · 2026-05-05T22:17:24Z

+ */
+package org.apache.kafka.clients.admin;
+
+public class KafkaAdminClientInternalFactory {


Exposing a package-private method through this 'factory' feels very hacky. I'll submit a PR to move the specific test from core to clients-integration-test so we can close this door for good.

chia7712 · 2026-05-05T22:19:39Z

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class FailureInjectingTimeoutProcessorFactory extends KafkaAdminClient.TimeoutProcessorFactory {


similar to KafkaAdminClientInternalFactory, which can be moved out of testfixtures.

chia7712 · 2026-05-05T22:23:49Z

+      config.dependencies.each { dep ->
+        if (dep instanceof FileCollectionDependency) {
+          dep.files.files.each { file ->
+            if (file.absolutePath.endsWith('/clients/build/classes/java/test')) {


Nice catch. Right now it only rejects client test code, but since other modules still depend on other's test code, this will eventually result in all test scopes being rejected as we clean them up. A beautiful world! 😄

apache/kafka#22201 (3b6c838) moved org.apache.kafka.test.TestUtils out of clients' regular test sources and into testFixtures, so the class is no longer in kafka-clients-<v>-test.jar but in the new kafka-clients-<v>-test-fixtures.jar instead. Add the test-fixtures classifier dependency so ClusterWaitTest compiles again. (Earlier attempt swapped the import to org.apache.kafka.common.test.TestUtils from kafka-test-common-runtime, but that class is package-private — its javadoc redirects callers to org.apache.kafka.test.TestUtils. CI confirmed this with: "TestUtils is not public in org.apache.kafka.common.test; cannot be accessed from outside package".) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Drop the dependency on org.apache.kafka.test.TestUtils#waitForCondition and inline an equivalent polling loop. Test scope only. apache/kafka#22201 (3b6c838) moved TestUtils out of kafka-clients' test classifier jar into testFixtures. Two attempts to keep using an upstream TestUtils failed: 1. Switching to org.apache.kafka.common.test.TestUtils (in kafka-test-common-runtime) — that class is package-private; CI reported "TestUtils is not public in org.apache.kafka.common.test; cannot be accessed from outside package". 2. Adding kafka-clients with classifier=test-fixtures so the original class is back on the test classpath — Confluent's CodeArtifact does not publish that classifier ("kafka-clients:jar:test-fixtures :8.4.0-81-ccs (absent)"). Inlining is small, self-contained, and robust to future upstream test-utility refactors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

chia7712 · 2026-05-09T05:15:34Z

    compileOnly libs.jacksonJDK8Datatypes
    compileOnly libs.jose4j          // for SASL/OAUTHBEARER JWT validation; only used by broker

+    testFixturesApi project(':test-common:test-common-util')


What do you think about using testFixturesImplementation instead? It provides better isolation for our dependencies.

Addressed in #22243

omkreddy · 2026-05-13T04:04:19Z

@soarez I see that we are not publishing kafka-clients-test-fixtures-4.4.0-SNAPSHOT.jar. I wanted to know if there is a specific reason for this. The reason I’m asking is that there are downstream projects that depend on these test utilities, so it would be useful to publish them.

cc @chia7712

chia7712 · 2026-05-13T04:56:27Z

@omkreddy thanks for pointing that out. The test-fixtures jar definitely needs to be published. We’ll file a patch for it shortly.

soarez · 2026-05-13T06:26:53Z

No, I didn’t think of downstream dependencies. It does make sense to publish it. And I see there’s already a patch! Thanks @majialoong

See #22201 (comment) . This change add `testFixturesJar` to the published artifacts. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ken Huang

chia7712 · 2026-06-29T10:58:00Z

+      dep instanceof ProjectDependency && dep.name == project.name
+    }
+    dependencies {
+      testFixturesApi files(sourceSets.main.output.classesDirs)


We should revisit those tests instead of hacking the dependencies. The test code in the clients module should NOT depend on io.opentelemetry, and that can be addressed by moving the io.opentelemetry-related code to ClientTelemetryUtils. It is not perfect, but it's better than a hack in build.gradle

See the discussion: #22201 (comment) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

chia7712 · 2026-07-18T22:59:35Z


    testImplementation project(':clients')
-    testImplementation project(':clients').sourceSets.test.output
+    testImplementation testFixtures(project(':clients'))


Seeing that tools depends on the testFixtures(project(':clients')), the testImplementation libs.bcpkix could be deleted from tools module.

We will file a minor patch for it

Hi @chia7712,
I've opened #22868 to address this. PTAL when you have time, thanks!

github-actions Bot added core Kafka Broker kraft build Gradle build or GitHub Actions clients labels May 4, 2026

rewire testfixtures deps to avoid shadow issues

846e889

soarez mentioned this pull request May 4, 2026

MINOR: extract TestSslUtils to new test-common-base #22193

Closed

chia7712 reviewed May 4, 2026

View reviewed changes

Clarify conflict comment and refer to shadow dep rewiring

913f688

chia7712 reviewed May 5, 2026

View reviewed changes

chia7712 mentioned this pull request May 5, 2026

KAFKA-19009: Move MetadataCacheTest to metadata module #22118

Merged

soarez added 2 commits May 5, 2026 11:15

rm all deps to clients test classpath

b3b5334

Disallow dependency on clients test classpath

559aec6

github-actions Bot added streams consumer labels May 5, 2026

mv additional classes

4e8fa3f

chia7712 approved these changes May 5, 2026

View reviewed changes

chia7712 changed the title ~~MINOR: Move TestSslUtils to client test fixtures~~ MINOR: Move client test utilities to test fixtures and resolve shadow jar conflicts May 5, 2026

chia7712 merged commit 3b6c838 into apache:trunk May 5, 2026
35 of 37 checks passed

KrishVora01 mentioned this pull request May 7, 2026

CPBR-3765: Fix ClusterWaitTest import for common-docker master builds (8.4.x + ) confluentinc/common-docker#1679

Merged

2 tasks

chia7712 reviewed May 9, 2026

View reviewed changes

majialoong mentioned this pull request May 13, 2026

MINOR: Publish kafka-clients test-fixtures artifact #22275

Merged

omkreddy pushed a commit that referenced this pull request May 13, 2026

MINOR: Publish kafka-clients test-fixtures artifact (#22275)

7a64fe8

See #22201 (comment) . This change add `testFixturesJar` to the published artifacts. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ken Huang

chia7712 reviewed Jun 29, 2026

View reviewed changes

m1a2st mentioned this pull request Jun 29, 2026

MINOR: Remove clients test-fixtures shadow JAR rewire hack #22699

Merged

chia7712 pushed a commit that referenced this pull request Jul 2, 2026

MINOR: Remove clients test-fixtures shadow JAR rewire hack (#22699)

cac4d1f

See the discussion: #22201 (comment) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

chia7712 reviewed Jul 18, 2026

View reviewed changes


		apply plugin: 'java-library'

		// Resolve conflict between testFixtures and shadow in :clients.

		@@ -1120,6 +1139,7 @@ project(':core') {
		implementation libs.re2j

		testImplementation project(':clients').sourceSets.test.output

Uh oh!

Conversation

soarez commented May 4, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omkreddy commented May 13, 2026

Uh oh!

chia7712 commented May 13, 2026

Uh oh!

soarez commented May 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

soarez commented May 4, 2026 •

edited by github-actions Bot

Loading