schema evolution test for kv store by lucliu1108 · Pull Request #4266 · confluentinc/schema-registry

Lucy Liu (lucliu1108) · 2026-04-17T17:07:26Z

What

Checklist

Please answer the questions with Y, N or N/A if not applicable.

[ ] Contains customer facing changes? Including API/behavior changes
[ ] Is this change gated behind config(s)?
- List the config(s) needed to be set to enable this change
[ ] Did you add sufficient unit test and/or integration test coverage for this PR?
- If not, please explain why it is not required
[ ] Does this change require modifying existing system tests or adding new system tests?
- If so, include tracking information for the system test changes
[ ] Must this be released together with other change(s), either in this repo or another one?
- If so, please include the link(s) to the changes that must be released together

References

JIRA:

Test & Review

Open questions / Follow-ups

confluent-cla-assistant · 2026-04-17T17:07:38Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

Copilot

Pull request overview

Adds an initial integration test suite to validate Avro schema evolution behavior when Kafka Streams materializes a KTable into a header-based KV state store (schema ID carried in record headers via HeaderSchemaIdSerializer).

Changes:

Introduces value schema evolution coverage (v1 → v2, add field with default) ensuring previously-written values remain readable.
Introduces key schema evolution coverage demonstrating that evolving the key schema results in distinct store entries (different key byte representations).
Adds shared test helpers to start a Streams KTable app with DSL_STORE_FORMAT_CONFIG=HEADERS, create topics, and wait for store population.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-17T17:11:12Z

+import org.apache.kafka.common.utils.Bytes;
+import java.util.stream.Collectors;
+import org.apache.kafka.streams.KafkaStreams;


Import ordering is inconsistent: java.util.stream.Collectors is placed after org.apache.kafka.common.utils.Bytes. This can trip import-order checks (and differs from the ordering used in KafkaStreamsHeaderSchemaIdIntegrationTest). Reorder/group imports so all java.* imports come before org.* imports.

Copilot · 2026-04-17T17:11:12Z

+      } catch (Exception e) {
+        // Store may not be ready yet
+      }


waitForStoreToContainKeys catches and ignores all exceptions. This can mask real failures (e.g., serialization/config errors) and turn them into a generic timeout. Catch only the expected transient exception for an unready store (e.g., InvalidStateStoreException) and either rethrow or record unexpected exceptions so the final assertion includes the root cause.

Copilot · 2026-04-17T17:11:12Z

+    builder.table(
+        inputTopic,
+        Consumed.with(createKeySerde(), createValueSerde()),
+        Materialized.<GenericRecord, GenericRecord, KeyValueStore<Bytes, byte[]>>as(STORE_NAME)
+            .withKeySerde(createKeySerde())
+            .withValueSerde(createValueSerde()));


startTableApp creates new key/value serde instances multiple times (for Consumed and again for Materialized). This duplicates config and can create multiple schema-registry client instances unnecessarily. Consider constructing the key/value serdes once and reusing them for both Consumed.with(...) and Materialized.with*Serde(...).

Suggested change

builder.table(

inputTopic,

Consumed.with(createKeySerde(), createValueSerde()),

Materialized.<GenericRecord, GenericRecord, KeyValueStore<Bytes, byte[]>>as(STORE_NAME)

.withKeySerde(createKeySerde())

.withValueSerde(createValueSerde()));

GenericAvroSerde keySerde = createKeySerde();

GenericAvroSerde valueSerde = createValueSerde();

builder.table(

inputTopic,

Consumed.with(keySerde, valueSerde),

Materialized.<GenericRecord, GenericRecord, KeyValueStore<Bytes, byte[]>>as(STORE_NAME)

.withKeySerde(keySerde)

.withValueSerde(valueSerde));

Alieh Saeedi (aliehsaeedii) · 2026-04-20T16:16:47Z

+
+      // Lookup with v1 key STILL finds the old entry (it was not overwritten)
+      ValueAndTimestamp<GenericRecord> resultV1Again =
+          store.get(newRecord(KEY_SCHEMA_V1, "sensorId", "sensor-1"));


Why not store.get(key)?

the whole newRecord(...) is just the key. I could rewrite that to a key variable to avoid creating the same key twice(one in producer, one during querying).

Alieh Saeedi (aliehsaeedii) · 2026-04-20T16:17:04Z

+          "v2 key lookup should return the value produced with v2 key");
+
+      // Lookup with v1 key STILL finds the old entry (it was not overwritten)
+      ValueAndTimestamp<GenericRecord> resultV1Again =


Why ValueAndTimestamp?

I switched it back to ValueTimestampHeaders type to align with other IQv1 tests.

add initial schema evolution test for kv store

1c54681

Copilot AI review requested due to automatic review settings April 17, 2026 17:07

Copilot started reviewing on behalf of Lucy Liu (lucliu1108) April 17, 2026 17:07 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

add key schema bytes identical test

07d6380

Lucy Liu (lucliu1108) changed the title ~~add initial schema evolution test for kv store~~ schema evolution test for kv store Apr 17, 2026

Alieh Saeedi (aliehsaeedii) reviewed Apr 20, 2026

View reviewed changes

Lucy Liu (lucliu1108) added 6 commits April 21, 2026 14:12

add more tests

4afdc1a

change method name

786bfa5

fix redundant key initialization

dfd9905

adjust test

dd49c30

add specificRecord evolution test

88ecd42

use v1 schema instead of generic one for old reader/writer

5ce8c40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schema evolution test for kv store#4266

schema evolution test for kv store#4266
Lucy Liu (lucliu1108) wants to merge 8 commits into
confluentinc:masterfrom
lucliu1108:schema-evolution-test

Lucy Liu (lucliu1108) commented Apr 17, 2026

Uh oh!

confluent-cla-assistant Bot commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Copilot AI Apr 17, 2026

Uh oh!

Alieh Saeedi (aliehsaeedii) Apr 20, 2026

Uh oh!

Lucy Liu (lucliu1108) Apr 21, 2026

Uh oh!

Alieh Saeedi (aliehsaeedii) Apr 20, 2026

Uh oh!

Lucy Liu (lucliu1108) Apr 21, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Lucy Liu (lucliu1108) commented Apr 17, 2026

What

Checklist

References

Test & Review

Open questions / Follow-ups

Uh oh!

confluent-cla-assistant Bot commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Alieh Saeedi (aliehsaeedii) Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Lucy Liu (lucliu1108) Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Alieh Saeedi (aliehsaeedii) Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Lucy Liu (lucliu1108) Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Lucy Liu (lucliu1108) Apr 21, 2026 •

edited

Loading