tests/rptest: expose L0 GC unreadability after a storage-mode flip by Lazin · Pull Request #31065 · redpanda-data/redpanda

Lazin · 2026-07-10T11:28:14Z

Reworks the storage-mode toggle ducktape test so it reproduces a data
unreadability bug in the mode-flip / L0-GC interaction (report:
https://gist.github.com/nvartolomei/fdb274b629a2184ec63709eb740d8c5d).

After a cloud → tiered_v2 storage-mode flip, the tiered read branch
serves everything at or above the local log's start offset from the local
log, ignoring the last-reconciled offset. The local log still holds
placeholders for the already-reconciled region, and L0 GC is allowed to
delete the objects those placeholders reference once their epoch goes
inactive and the minimum object age passes. Reading that region then has
to materialize placeholders against deleted objects, which fails with
download_not_found and no L1 fallback — even though L1 holds a complete
copy of the data. The flip also freezes local prefix-truncation
(tiered_v2 honors full topic retention), so the log never self-heals by
trimming the placeholder region.

The test shrinks cloud_topics_short_term_gc_minimum_object_age (12h →
30s) and the epoch/housekeeping intervals that gate GC eligibility,
disables the batch cache, ends the toggling in tiered_v2,
trickle-produces until the reconciled L0 objects are collected, wipes the
cloud-storage disk cache with a rolling restart, and reads the whole log
back. The read wedges on the placeholder region (~94k
Failed to materialize sorted run: download_not_found warnings in a
local run) instead of returning every acked record.

This PR is a reproducer: the altered test is expected to FAIL until the
bug is fixed. It is opened as a draft to anchor the discussion of the
fix (make the tiered read branch respect the last-reconciled offset, fall
back to L1 on materialization failure, or resume prefix-truncation of the
reconciled region after a flip).

Depends on #30966 (the storage-mode impl rework), whose commits are
included underneath.

Backports Required

Release Notes

none

Switch the flag from explicit_only to always so it activates as soon as every node runs v26.2, with no admin action. The create and alter-config gates keep rejecting cloud-architecture tiered topics (and cloud -> tiered_cloud conversions) while the cluster is only partially upgraded, and open automatically once the upgrade completes. set_feature_active now tolerates always-policy features jumping straight from unavailable to active without an observable available state, and the real-upgrade finalization test expects the flag to auto-activate after finalize. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>

The set_min_allowed_local_threshold stm command is new in v26.2, so the compaction sink must not replicate it while pre-v26.2 replicas may still be in the raft group. Gate the notifier on the tiered_cloud_topics feature flag: until it activates (the cluster is fully upgraded to v26.2) every notification attempt reports success without replicating. This is safe because no cloud-architecture tiered topic -- the only reader of local data below the floor -- can exist while the flag is inactive. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>

The tiered_cloud storage mode ships in v26.2 as the second implementation of tiered storage, so the user surface is reworked around the 'tiered' name instead of exposing the internal enum: - redpanda.storage.mode accepts unset/local/tiered/cloud; the 'tiered_cloud' spelling is no longer a valid value. Both tiered variants display as 'tiered' in DescribeConfigs. - The new default_redpanda_storage_mode_tiered_impl cluster config (tiered_v1 = classic Tiered Storage, tiered_v2 = Cloud Topics based) selects which implementation 'tiered' refers to. It defaults to tiered_v1 for all clusters, so an upgrade never changes what 'tiered' means and tiered_v2 is an explicit opt-in. - The new read-only redpanda.storage.mode.impl topic property reports the exact implementation of every topic's storage mode (unset/local/tiered_v1/tiered_v2/cloud), making the describe output unambiguous regardless of the reader's cluster config. On creation it selects the storage mode precisely, alone or together with a matching redpanda.storage.mode; mismatched pairs are rejected. The property is not alterable, although sets that match the current value are tolerated as no-ops so that describe-then-alter round trips (which replay every config) keep working. It is reported with the default config source: the value is derived from the storage mode rather than being an explicit override, so config backup/replay tooling must not treat it as user-set. Storage-mode transition errors print the impl names, so a rejected tiered_v1 to tiered_v2 transition does not read as 'tiered to tiered'. The tiered_v2 impl keeps the tiered_cloud_topics feature and cloud storage gates. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>

The synced describe output shows 'tiered' for both tiered impls, so the mode string alone is ambiguous. Request the read-only redpanda.storage.mode.impl property alongside the mode and apply it first (it is exact and reported for every topic), reconstructing the source topic's storage mode on the target instead of re-interpreting 'tiered' through the target's own tiered-impl config - with different configs on the two clusters that could silently turn a cloud-architecture topic into a classic tiered one and start an archiver on a partition that has no archival stm. The mode string remains as a fallback for sources that predate the impl property. Mirror-topic creation and property updates bypass the kafka handlers, so this path enforces the guards itself: storage-mode transition rules (a bad sync becomes a rejected update instead of an applied forbidden transition) and the tiered_cloud_topics upgrade gate - no mirror topic is created with or moved to the tiered_v2 impl until the cluster is fully upgraded to v26.2; creation is skipped and retried once the feature activates. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>

TieredCloudUpgradeTest checks the mixed-version behavior: creating a tiered_v2 topic or converting a cloud topic to it is rejected (or, via a v26.1 controller that drops the unknown impl property, degrades to a classic tiered topic) while the cluster is partially upgraded, and both operations work once every node runs v26.2. StorageModeAliasMatrixTest covers every mode/impl input combination under both tiered impls, including impl-only creation, mode/impl consistency rejections, read-only enforcement and the always-present describe entry. ShadowLinkStorageModeSyncTest syncs cloud and tiered topics across all four combinations of source/target tiered impls, including a mid-flight cloud to tiered_v2 conversion. TopicSpec spells tiered variants through the new storage_mode_config helper (mode=tiered plus the impl property, which keeps v26.1 brokers on a tiered mode instead of silently falling back to the cluster default), kafka-cli describe output round-trips through the new impl kwarg, and the describe-topics parser accepts the asterisk kcl appends to read-only properties. The storage-mode toggle and rnot flipping tests point the 'tiered' alias at tiered_v2 explicitly, since the alter path resolves it through the cluster config. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>

Rework the storage-mode toggle test to reproduce the mode-flip / L0-GC interaction: after a cloud -> tiered_v2 flip the tiered read branch serves everything at or above the local log start from the local log, ignoring the last-reconciled offset. The local log still holds placeholders for the already-reconciled region, L0 GC deletes the objects they reference once their epoch goes inactive and the minimum age passes, and reading the region then fails to materialize with no L1 fallback - even though L1 holds a complete copy. The flip also freezes local prefix-truncation, so the log never self-heals. The test shrinks the GC minimum object age and the epoch/housekeeping intervals that gate GC eligibility, disables the batch cache, ends the toggling in tiered_v2, trickle-produces until the reconciled L0 objects are collected, wipes the cloud-storage disk cache with a rolling restart and then reads the whole log back. The read currently wedges on the placeholder region (download_not_found materialization failures) instead of returning every acked record. The test is expected to FAIL until the read path falls back to L1 (or prefix-truncation trims the placeholder region) after a mode flip. Signed-off-by: Evgeny Lazin <4lazin@gmail.com>

Copilot

Pull request overview

This PR reworks storage-mode configuration to introduce an explicit, unambiguous tiered implementation selector (tiered_v1 vs tiered_v2) via a new read-only topic property redpanda.storage.mode.impl, updates cluster-linking to sync that property safely across clusters with different defaults, and adjusts ducktape coverage (including a draft reproducer expected to fail) to expose an L0-GC unreadability after a cloud -> tiered_v2 flip.

Changes:

Add default_redpanda_storage_mode_tiered_impl cluster config and redpanda.storage.mode.impl topic property; update create/describe/alter validation and parsing to use the new (mode, impl) semantics.
Update cluster-linking topic reconciler and property sync to preserve tiered variant correctly across differing defaults and enforce upgrade gating for tiered_v2.
Update/extend rptest coverage and add an end-to-end mode-flip/L0-GC reproducer (expected to fail until the underlying bug is fixed).

Reviewed changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/rptest/tests/topic_creation_test.py	Normalize kcl output by stripping `*` from read-only config keys.
tests/rptest/tests/tiered_cloud_local_retention_test.py	Switch tiered-cloud usage to tiered_v2 via `storage_mode_config`.
tests/rptest/tests/storage_mode_test.py	Add upgrade-gate + alias/impl matrix coverage for new storage mode semantics.
tests/rptest/tests/shadow_linking_rnot_test.py	Pin tiered default to v2 for flip workload; use `storage_mode_config`.
tests/rptest/tests/random_node_operations_smoke_test.py	Use `storage_mode_config` for tiered_v2 topic creation.
tests/rptest/tests/follower_fetching_test.py	Use `storage_mode_config` for tiered_v2 fetch-from case.
tests/rptest/tests/describe_topics_test.py	Document/parse new `redpanda.storage.mode.impl` (and optional `*` marker).
tests/rptest/tests/cluster_linking_test_base.py	Sync `redpanda.storage.mode.impl` and update storage-mode sets used by tests.
tests/rptest/tests/cluster_linking_storage_mode_sync_test.py	New test verifying storage-mode variant survives sync across differing defaults.
tests/rptest/tests/cluster_linking_e2e_test.py	Update cloud-topic replication tests to use tiered_v2/impl semantics.
tests/rptest/tests/cluster_features_test.py	Adjust tiered_cloud_topics expectations for `available_policy::always`.
tests/rptest/tests/cloud_topics/retention_test.py	Use tiered_v2 storage-mode config helper in retention tests.
tests/rptest/tests/cloud_topics/iceberg_test.py	Use tiered_v2 storage-mode config helper in Iceberg tests.
tests/rptest/tests/cloud_topics/e2e_test.py	Rework mode-toggle test into L0-GC unreadability reproducer (expected fail).
tests/rptest/tests/cloud_topics/debug_rows_test.py	Update test matrix to include tiered_v2.
tests/rptest/tests/cloud_topics/compaction_stress_test.py	Update tiered-cloud references to tiered_v2/impl.
tests/rptest/tests/cloud_topics_test.py	Update tiered-cloud references to tiered_v2/impl.
tests/rptest/services/redpanda.py	Allow `await_feature` to accept `{available,active}` for always-auto-active features.
tests/rptest/clients/types.py	Add `storage_mode_config()` helper + `PROPERTY_STORAGE_MODE_IMPL`.
src/v/model/tests/storage_mode_alias_test.cc	New unit tests for user/impl parsing and naming behavior.
src/v/model/tests/BUILD	Register new storage_mode_alias gtest target.
src/v/model/model.cc	Implement tiered-impl parsing + user-vs-impl vocabulary and naming helpers.
src/v/model/metadata.h	Define `redpanda_storage_mode_tiered_impl` and helper APIs.
src/v/kafka/server/tests/alter_config_test.cc	Ensure new `redpanda.storage.mode.impl` appears in describe-configs coverage.
src/v/kafka/server/handlers/topics/validators.h	Extend create-topic validation for `(mode, impl)` semantics.
src/v/kafka/server/handlers/topics/types.cc	Prefer impl property over alias-resolved mode when building topic configs.
src/v/kafka/server/handlers/incremental_alter_configs.cc	Enforce impl read-only; parse storage mode via user-string semantics.
src/v/kafka/server/handlers/create_topics.cc	Allow `redpanda.storage.mode.impl` as supported create-topic config.
src/v/kafka/server/handlers/configs/config_utils.h	Improve transition error messages to use impl names.
src/v/kafka/server/handlers/configs/config_response_utils.cc	Describe `storage.mode` as user name; add read-only `storage.mode.impl`.
src/v/kafka/server/handlers/alter_configs.cc	Enforce impl read-only; parse storage mode via user-string semantics.
src/v/kafka/protocol/topic_properties.h	Define constant for `redpanda.storage.mode.impl`.
src/v/features/feature_table.h	Change tiered_cloud_topics to `available_policy::always` (auto-activate).
src/v/config/rjson_serialization.h	Add JSON serialization for tiered-impl enum.
src/v/config/rjson_serialization.cc	Implement JSON serialization for tiered-impl enum.
src/v/config/property.h	Treat tiered-impl enum as string-typed property.
src/v/config/convert.h	Add YAML conversion for tiered-impl enum.
src/v/config/configuration.h	Add `default_redpanda_storage_mode_tiered_impl` cluster config binding.
src/v/config/configuration.cc	Define `default_redpanda_storage_mode_tiered_impl` with docs and defaults.
src/v/cluster_link/utils/topic_properties_utils.h	Add helper to combine `(mode, impl)` into a safe storage-mode update.
src/v/cluster_link/utils/topic_properties_utils.cc	Implement `(mode, impl)` combination + gating + transition validation.
src/v/cluster_link/utils/BUILD	Link cluster_link utils against features for gating logic.
src/v/cluster_link/topic_reconciler.h	Inject feature_table into reconciler for upgrade gating and mode updates.
src/v/cluster_link/topic_reconciler.cc	Apply storage mode from `(mode, impl)` pair and gate tiered_v2 on upgrades.
src/v/cluster_link/tests/topic_reconciler_test.cc	Add tests for gating behavior in storage-mode updates.
src/v/cluster_link/tests/link_test.cc	Update manager construction for new feature_table dependency.
src/v/cluster_link/tests/deps.h	Add feature_table to test fixture wiring.
src/v/cluster_link/tests/deps.cc	Start/activate feature_table in test fixture wiring.
src/v/cluster_link/tests/BUILD	Add features + topic_properties_utils deps for cluster_link tests.
src/v/cluster_link/service.cc	Pass controller feature_table into cluster_link manager.
src/v/cluster_link/model/types.h	Sync `redpanda.storage.mode.impl` in default-synced topic properties.
src/v/cluster_link/manager.h	Plumb feature_table pointer into manager.
src/v/cluster_link/manager.cc	Plumb feature_table pointer into topic_reconciler construction.
src/v/cloud_topics/level_zero/notifier/tests/level_zero_notifier_test.cc	Add feature-gated no-op coverage for notifier calls pre-upgrade.
src/v/cloud_topics/level_zero/notifier/tests/BUILD	Add features dependency for notifier tests.
src/v/cloud_topics/level_zero/notifier/level_zero_notifier.h	Gate notifications on tiered_cloud_topics feature activation.
src/v/cloud_topics/level_zero/notifier/level_zero_notifier.cc	Implement gating logic (no-op until feature active).
src/v/cloud_topics/level_zero/notifier/BUILD	Add features dependency for notifier component.
src/v/cloud_topics/app.cc	Pass controller feature_table to the L0 notifier.

+// Read-only companion of redpanda.storage.mode: the resolved variant
+// (tiered_v1/tiered_v2) of a topic in the tiered storage mode. Accepted on
+// topic creation (only together with redpanda.storage.mode=tiered) to pick a
+// non-default variant; not alterable.


+ * The optional redpanda.storage.mode.impl property picks the tiered
+ * variant and is only valid together with redpanda.storage.mode=tiered.


+    if (
+      cfg.properties.storage_mode
+        == ::model::redpanda_storage_mode::tiered_cloud
+      && !_feature_table->local().is_active(
+        features::feature::tiered_cloud_topics)) {


+    try {
+        if (
+          utils::maybe_append_storage_mode_update(
+            update,
+            find_config(kafka::topic_property_redpanda_storage_mode),
+            find_config(kafka::topic_property_redpanda_storage_mode_impl),
+            local_topic_config,
+            _feature_table->local())) {
+            config_updated = true;
+        }
+    } catch (const std::exception& e) {


vbotbuildovich · 2026-07-10T13:14:45Z

Retry command for Build#86956

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/cloud_topics/e2e_test.py::EndToEndCloudTopicsStorageModeToggleTest.test_toggle_storage_mode

vbotbuildovich · 2026-07-10T13:32:12Z

CI test results

test results on build#86956

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FAIL	EndToEndCloudTopicsStorageModeToggleTest	test_toggle_storage_mode	null	integration	https://buildkite.com/redpanda/redpanda/builds/86956#019f4bf7-3934-4b31-bb5e-ba52cfdfb73c	0/11	Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsStorageModeToggleTest&test_method=test_toggle_storage_mode
FAIL	EndToEndCloudTopicsStorageModeToggleTest	test_toggle_storage_mode	null	integration	https://buildkite.com/redpanda/redpanda/builds/86956#019f4bfb-d7fe-4fc1-b5ba-86d27160d802	0/11	Test FAILS after retries.Significant increase in flaky rate(baseline=0.0000, p0=0.0000, reject_threshold=0.0100)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsStorageModeToggleTest&test_method=test_toggle_storage_mode

Lazin added 6 commits July 9, 2026 11:05

github-actions Bot added area/build area/redpanda labels Jul 10, 2026

Lazin marked this pull request as ready for review July 10, 2026 12:11

Lazin requested a review from a team as a code owner July 10, 2026 12:11

Copilot AI review requested due to automatic review settings July 10, 2026 12:11

Copilot started reviewing on behalf of Lazin July 10, 2026 12:11 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tests/rptest: expose L0 GC unreadability after a storage-mode flip#31065

tests/rptest: expose L0 GC unreadability after a storage-mode flip#31065
Lazin wants to merge 6 commits into
redpanda-data:devfrom
Lazin:ct/potato-flag-4

Lazin commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

vbotbuildovich commented Jul 10, 2026 •

edited

Loading

Uh oh!

vbotbuildovich commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		* The optional redpanda.storage.mode.impl property picks the tiered
		* variant and is only valid together with redpanda.storage.mode=tiered.

Uh oh!

Conversation

Lazin commented Jul 10, 2026

Backports Required

Release Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

vbotbuildovich commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Retry command for Build#86956

Uh oh!

vbotbuildovich commented Jul 10, 2026

CI test results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vbotbuildovich commented Jul 10, 2026 •

edited

Loading