RFC ts-ct-migration: enable + e2e suite [PR 8]#30896
Draft
sjust-redpanda wants to merge 58 commits into
Draft
Conversation
This was referenced Jun 23, 2026
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr8
branch
4 times, most recently
from
99af7cd to
9f3904c
Compare
In order to dynamically switch a partition from TS to CT/TSv2, we need to be able to enable the ctp_stm on the fly. The simplest way to do that is to simply attach an empty ctp_stm to every partition. Unfortunately, with LRO unset, ctp_stm previously returned offset::min() for get_max_collectible_offset() preventing log trimming. This commit modifies that behavior to return offset::max() if _max_applied_epoch is empty making the presence of a ctp_stm harmless until advance_epoch() is called. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tions is_applicable_for: pre-install the (idle) ctp_stm on tiered-storage partitions when cloud topics are available, so a tiered->cloud migration needs no runtime STM install -- the STM is already present and inert until the partition becomes a cloud topic. STM membership is fixed at construction, so it has to be there beforehand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…verload Split advance_reconciled_offset so the reconciled log offset can be supplied explicitly instead of always deriving it from the raft offset translator. The existing kafka-offset-only entry point now delegates, deriving the log offset as before. This lets a caller seed the reconciliation baseline of a TS-migrated partition directly to the migration boundary -- passing the precise raft offset of the last uploaded tiered-storage segment -- so the already reconciled region can be trimmed without re-deriving the offset. ctp_stm carries no migration state of its own; this is expressed purely as "reconciliation starts at offset N". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
During migration, we need the newly configured TSv2 or CT partition to continue to behave as an archival stm. Thus, we need the archival stm to continue to exist after the config switch. The simplest way to do this is to allow it to exist generally on CT/TSv2 topics. Drop the cloud_topic_enabled() == false guard from archival_metadata_stm_factory::is_applicable_for, so every cloud-topic partition carries an archival STM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cloud-mode partition whose storage mode has been flipped reports is_archival_enabled() == false, which would make max_removable_local_log_offset collect_all and stop constraining local-log truncation -- evicting tiered-storage data not yet uploaded. Keep constraining via cloud_recoverable_offset() while the partition still holds archived data (a non-empty live manifest or a spillover archive), introduced here as holds_archived_data(); once the manifest is emptied, trimming resumes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the cloud_topic_enabled() guard from log_eviction_stm_factory so the eviction STM is installed on cloud-topic partitions too. A partition migrating from tiered storage is still served from its local log + archival manifest and must keep supporting prefix truncation (DeleteRecords) while migrating; the trigger flips storage_mode to cloud without reconstructing the partition, and a still-migrating partition is reconstructed cloud-mode on restart/recovery, so the STM must already be present. On a native cloud topic it is an inert passenger. Subsequent integration tests will provide coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
remote_segment_index.{h,cc} contained two independent things sharing a TU: the
read/write `offset_index` (used by readers) and `remote_segment_index_builder`, a
storage::batch_consumer that pulls in raft (raft::offset_translator_batch_types).
offset_index itself has none of that coupling -- only model, bytes, utils/delta_for
and serde.
Move `offset_index` into cloud_storage/offset_index.{h,cc} behind a new
//src/v/cloud_storage:offset_index cc_library.
remote_segment_index.{h,cc} keeps segment_record_stats + the builder, now
including offset_index.h; the cloud_storage monolith depends on :offset_index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional descriptor that lets an L1 object/extent point at a segment that physically lives in the tiered-storage bucket instead of an L1-native object. Metadata/encoding only; no read, write, or delete path consumes it yet. - imported_ts_info (object_id.h): the per-segment descriptor -- ts_path, delta_offset/delta_offset_end (log<->kafka offset translation), segment_term, and last_kafka_offset. Stored split by ownership: ts_path on the object row (imported_ts_object_location), delta/term on the extent row (imported_ts_segment_info); absent means a native L1 object. - Thread it through the metastore read/interface types on both backends: an optional imported_ts_info on object_response / extent_object_info (recomposed from the storage split on read), plus rpc_types and the replicated/domain passthrough. - Encode it: serde-versioned on both backends (object_entry/extent and the LSM rows), the LSM debug-serde path, and the ImportedTsObjectLocation / ImportedTsSegmentInfo protobufs. Old-version decode defaults to absent. Testing: serde round-trip unit tests for the new fields, including old-version decode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a durable, per-partition boolean migrating to each backend's partition record, set true while a partition is mid tiered->cloud migration. - state: add the field to partition_state (serde v0->v1) and the LSM metadata_row_value (v1->v2), defaulting to false so snapshots predating the field decode as native cloud topics; carry it through the MetadataValue debug proto. - op: add set_migrating(partition, bool) across the metastore interface, the RPC service/client, the leader router (+probe), both domain managers, and both backends. It is idempotent (setting the current value is a no-op), applies in either direction, and creates the partition record when set on an absent partition; metadata-row rewrites (add_objects, set_start_offset) preserve the flag. - read: surface the flag on get_offsets. Tests (both backends): create-on-absent, set-then-clear, idempotent set, and preservation across add_objects; serde back-compat for both records. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Let the metastore register tiered-storage segments as imported L1 extents
by reference, reusing the existing add_objects extent-addition path rather
than a dedicated command.
- add_objects now handles imported objects:
- skips the pre-registration check for imported objects (they reference an
existing tiered-storage segment; nothing is written to L1)
- copies imported_ts_location / imported_ts_delta onto the object and
extent rows
- sets start_offset from the first extent for an empty migrating partition
(a non-zero start -- the TS->CT mount), rather than requiring offset 0
- the object builder's add_imported(ntp_metadata) registers one imported
segment as a finished, single-extent object: with no L1 write to reserve it
creates the object id and records it as finished without pre-registration,
to be committed by a subsequent add_objects. meta.imported must be set; the
read-replica metastore rejects it.
Tests (both backends): a migrating partition adopts a non-zero start at the
first imported extent; successive batches forward-append; a non-contiguous
batch is rejected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist a partition-local storage mode (model::redpanda_storage_mode) in the partition properties STM, alongside the existing writes_disabled property. This is the partition's own durable record of its storage mode, decoupled from the topic config; a later change exposes it on ntp_config as partition_mode (vs the topic-config-derived topic_mode) so the TS->CT migration can be modelled as reconciling partition_mode toward topic_mode. Adds update_partition_mode_cmd / operation_type::update_partition_mode, carries partition_mode on state_snapshot and raft_snapshot (serde version bump to v2, compat_version unchanged so older snapshots decode with partition_mode = unset), and the set_partition_mode / partition_mode / sync_partition_mode API. apply is refactored to dispatch on operation_type and copy-modify the current snapshot so each property update preserves the others. partition_mode updates are idempotent by value and ordered by log offset (no per-property revision). Also adds a change callback (set_partition_mode_change_callback), fired when partition_mode changes on apply or on raft-snapshot restore, so an owner can re-read partition_mode() and propagate it (the STM had no change notification; writes_disabled is polled). Tests cover: default unset, set/idempotent/change with cross-node consistency; partition_mode and writes_disabled are independent on the shared snapshot; raft snapshot carries partition_mode at the right offsets; recovery (local kvstore snapshot + replay, and raft-snapshot recovery after dropping a node's data); and the change callback fires on a real change but not on an idempotent no-op. Inert: no callers yet (the command is wired up and gated behind a feature in follow-up commits). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Declare feature::tiered_to_cloud_migration (bit 18, explicit_only, v26_2_1). It gates all behaviors of the TS->CT migration capability across the PR stack: the partition_mode write introduced in this base branch, and the migration trigger/cutover in the later PRs. explicit_only means it does not auto-activate and carries no mixed-version compatibility commitment until an operator opts in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…mode
Introduce two storage-mode accessors on ntp_config:
- topic_mode() the mode the topic config requests (the existing
_overrides->storage_mode; the "desired" mode).
- partition_mode() the partition's own durable mode, fed from
partition_properties via set_partition_mode(); the
"actual" mode. Falls back to topic_mode() when unset.
The serving/behavior predicates (is_archival_enabled, is_remote_fetch_enabled,
is_tiered_storage, cloud_topic_enabled, is_tiered_cloud) now key on
partition_mode() instead of reading _overrides->storage_mode directly. The
legacy shadow_indexing fallback (used when the mode is unset) is preserved
unchanged, since archival-only/fetch-only SI modes have no redpanda_storage_mode
equivalent.
TS->CT migration wil lbe modelled as partition_mode lagging topic_mode
(still `tiered` while the topic config says `cloud`) until cutover, with
serving keyed on the actual mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ntp_config On partition start, read partition_properties_stm::partition_mode() and push it into the log's ntp_config (storage::log gains a set_partition_mode method, mirroring set_overrides), and register the STM's change callback so the value is re-pushed whenever the STM applies a partition_mode update or restores a raft snapshot. ntp_config::partition_mode() therefore tracks the partition's durable mode, falling back to topic_mode() while unset. No feature guard here: reading replicated STM state and pushing it into the local ntp_config is always safe. Writing partition_mode (which requires the tiered_to_cloud_migration feature) is added in the bootstrap commit; until then the value is always unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the partition's durable storage mode (partition_mode) in
partition_properties. It is populated:
- at partition creation, from the topic-config mode, when the migration
feature is active (partition_manager::manage);
- on becoming leader, for a partition that predates this (maybe_bootstrap_
partition_mode); a node that became leader after an in-place ts->ct switch
keys on the archival manifest (holds_archived_data) since topic_mode()
already reads cloud;
- on migration-feature activation, sweeping current leaders that were elected
while the feature was inactive.
The creation-time classification (set_creation_partition_mode) is preferred by
the bootstrap over re-reading topic_mode(), so the mode does not depend on the
async config propagation a runtime re-derivation would race.
Also add partition_properties_stm::seed_partition_mode(), which pre-writes the
kvstore-backed local snapshot before the STM starts so a partition restored with
a non-empty log comes up with partition_mode already set (the seed survives
start(), whose initial cleanup only runs for an empty log). Used by the
migration-recovery path.
partition_mode stays inert here; serving/behavior accessors begin keying on it
in later PRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
While a partition is migrating (ntp_config::is_migrating() -- the topic requests cloud but partition_mode still lags at tiered), the archiver dark-copies its tiered-storage manifest into the cloud-topics L1 metastore via an injected sink, trimming the front and appending the tail so a converged partition can cut over. The mirror and migration head-prune run only while is_migrating(). Because the mirror imports by trim-front + append-tail and cannot reflect an in-place segment rewrite, two manifest-mutating jobs are suspended while is_migrating(): compacted reupload and adjacent segment merging. Both key on is_migrating() rather than the storage mode: partition_mode lags at tiered during migration, so cloud_topic_enabled()/is_archival_enabled() report the tiered values and would let these jobs run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR2 pre-installs an idle ctp_stm on tiered-storage partitions, so the housekeeper now manages partitions that are not (yet) cloud topics -- a plain tiered partition, or one still migrating tiered->cloud (partition_mode lags at tiered until cutover). Running CT housekeeping against such a partition forces epoch/placeholder maintenance that seeds a meaningless reconciled offset and pins local-log GC, freezing retention. Gate the loop on cloud_topic_enabled() so it runs only on cloud topics, skipping until cutover. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…opic
When the mirror has covered the manifest tail, cut the partition over. The
coordination lives on the partition (it owns all three STMs); the archiver only
detects convergence and triggers it. partition::cutover_to_cloud_topic runs the
three STMs in order:
1. ctp_stm: seed the reconciliation baseline at boundary B (+ the DeleteRecords
trim floor), handing over the local-log trim floor.
2. partition_properties_stm: advance partition_mode tiered->cloud. This is the
routing flip -- the read/write path keys on cloud_topic_enabled(), which
reads partition_mode. ctp_stm is seeded first, so the flip never exposes
"neither STM authoritative"; and it must precede the manifest reset, else a
reader (partition_mode still tiered) would route to the emptied manifest.
3. archival_metadata_stm + L1 metastore: empty the manifest (releases the
local-trim clamp and makes the now-cloud-mode archiver dormant) and mark the
offline migration phase complete.
A crash after step 2 but before step 3 leaves partition_mode==cloud (routing
already correct) with a non-empty manifest that no archiver would clean up;
maybe_finish_cutover finishes the tail on the next leader (idempotent).
Dormancy and the reconciler/housekeeper skip drop the archival migration flag:
dormancy keys on cloud_topic_enabled() && !holds_archived_data(); the reconciler
skips a partition until it is a cloud topic (cloud_topic_enabled()), picking it
up the round after cutover advances partition_mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cluster metadata recovery bootstraps a cloud topic from the restored L1 metastore. Whether that is the right thing to do depends on the partition's migration phase, which the recovery backend reads from the metastore offsets response. Mid-migration (tiered->cloud): the authoritative data is in tiered storage and the L1 mirror is incomplete, so recover as tiered storage. The recovery backend sets the topic's recovery flag; controller_backend emplaces the remote_topic_properties for a recovering cloud-topic partition; and ntp_config::recovery_enabled() honors the override for cloud topics. The partition rebuilds its archival STM from the remote manifest and is served as tiered storage -- partition_mode lags at tiered, so cloud_topic_enabled() is false and reads route to the local log + manifest -- and the live mirror resumes and cuts over. partition_mode must load as `tiered` for the recovered partition, but the leadership bootstrap cannot classify this reliably: the archival manifest rebuilds asynchronously, so holds_archived_data() may still read false here and would durably misrecord `cloud` (the bootstrap is one-shot). Instead partition_manager::manage() pre-writes the partition_properties snapshot to `tiered` for a recovering cloud-topic partition. The recovered log is non-empty, so the seeded snapshot survives perform_initial_cleanup and is loaded when the STM starts -- partition_mode is `tiered` before leadership, and the bootstrap finds it already set and does nothing. Completed (or never migrated): L1 is authoritative. Cutover emptied the live archival manifest, but the remote manifest still holds the pre-cutover segments. Drop the leftover remote_topic_properties so the recovered archival STM does not re-adopt that manifest (partition_mode==cloud already routes to the cloud-topic path, so this is defence-in-depth; it also avoids re-adopting a stale manifest). Imported extents carry their own tiered-storage path (imported_segment_info::ts_path), so reads do not need these properties. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A read replica of a still-migrating source must not be misled by the source's incomplete L1 state. Read the source's migration_phase from the snapshot and carry it on the snapshot handle. No separate tiered-storage read path is needed: the cutover-last mirror copies the migrating source's tiered-storage data into L1 as imported extents, so the existing L1 read path serves the source through migration (lagging the source tail until cutover) and natively afterwards -- the imported mirror unifies the read path. The phase is surfaced for observability and to gate behavior at the cutover boundary. Also updates the recovery/read-replica e2e docstrings to reflect that E1/E2 are implemented build-green (their bodies, the validation vehicles, are not yet filled in / run). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Permit the redpanda.storage.mode transition tiered -> cloud / tiered_cloud (the migration trigger) only when the tiered_to_cloud_migration cluster feature is active, in both the alter_configs and incremental_alter_configs handlers. Previously those transitions were rejected outright, so a migration could not be triggered. Gating them keeps a mixed-version cluster from beginning a migration that older nodes cannot drive. The feature itself is added earlier in the stack (the partition_mode substrate), so this commit only wires the trigger gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A cutover-last migration e2e: produce in tiered mode, trigger the migration (storage-mode flip) mid-stream, and verify a consumer reading from offset 0 sees every record in order across the background mirror and the cutover (observed as the archival manifest going empty). Matrixed over cloud / tiered_cloud, and enables the tiered_to_cloud_migration feature. The transactions case is ported from the phase-1 e2e because read_committed correctness across the migration boundary is exactly what the imported reader must get right: committed + aborted transactions produced in tiered mode (aborted ranges sourced from the per-segment tx_range_manifest in S3 after _rm_stm drops its state), a transaction straddling the trigger, and post-cutover transactions -- a read_committed consumer must see exactly the committed records. Recovery and read-replica variants are present as stubs; their bodies are to be filled alongside E1/E2 (the offline consume-path routing) which they validate. Type-checks at strict level; behavioral validation requires a ducktape run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Exercises E1 end to end: a partition still mid tiered->cloud migration is recovered as tiered storage (its archival STM rebuilt from the remote manifest), not bootstrapped from the incomplete L1 mirror. A long-running producer keeps the partition migrating across the snapshot + wipe (cutover needs the producer to stop first); after the migrating state reaches object storage the cluster is abruptly stopped, wiped, and recovered. A consumer from offset 0 then sees a correct, gap-free prefix of the data that reached tiered storage (records acked but not yet uploaded are legitimately lost on a cluster wipe, so the exact total is nondeterministic). Verified: the recovery backend logs "Recovering mid-migration cloud topic ... as tiered storage", the recovered topic gets recovery_enabled + remote_topic_properties at the original revision, the archival STM rebuilds from the remote manifest, and the consumer reads ~114k contiguous records (invalid_reads=0, offset_gaps=0) with no crash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Exercises E2 end to end: a read replica of a partition still mid tiered->cloud migration serves the source's data through its L1 imported extents. A long-running producer keeps the source migrating while a second (read-replica) cluster, pointed at the source's bucket, reads it; a consumer from offset 0 sees a correct, gap-free prefix (~116k records, invalid_reads=0, offset_gaps=0). This is what surfaced the snapshot-metastore gap folded into the E2 commit: the replica read imported extents at the native L1 object path (404) until the snapshot path was taught to carry imported_segment_info. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Migrates a compact,delete topic mid-stream: phase-1 keyed records + tombstones are archived (and compacted) on tiered storage; phase-2 over the same key set is written after the trigger. After cutover the whole log lives in L1 (imported extents + native CT) and CT (L1) compaction runs over it; a read from offset 0 with latest-value validation must resolve every key to the producer's last value -- catching any key resurrected across the boundary or tombstone mishandled. Waits for CT compaction to quiesce (records-removed metric stable) before validating, since latest-value validation only holds once the log is fully compacted. Confirmed CT compaction converges across the unified imported+native log (cleaned ranges span the pre-cutover imported region from offset 0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A rate-limited producer runs continuously through the migration trigger and keeps writing; the partition must cut over to cloud topics while the producer is still active -- migration does not require the workload to quiesce. A consume from offset 0 must return every acked record contiguously across the mid-stream cutover (data above the manifest tail is reconciled into L1 from the raft log; writes after cutover take the cloud-topic path). It also asserts cutover is terminal: post-cutover produce must go to the cloud-topic write path (the L1 next_offset advances to cover it) and the archival manifest must stay empty (the archiver is dormant, not re-uploading raft-log writes and re-triggering the migration). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Exercises the mirror's head-prune (removal_mode::detach): retention.bytes trims the archival manifest head while the partition is still migrating, so the start offset advances above 0 with the manifest still non-empty. The mirror must drop the imported L1 extents below the new start without deleting the backing tiered-storage objects (the archiver still owns them during migration). After cutover the partition's start is the trimmed start (> 0) and a consumer from the earliest offset sees a correct, gap-free prefix -- no stale extents below the start, no gap. Verified: ~2MB of ~19MB produced is retained (head trimmed from ~150k to ~10k records), start carries through cutover, gap-free consume. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A 4-partition topic migrates on one storage-mode trigger: each partition runs its own archiver mirror and cuts over independently. Waits for every partition's TS segment before triggering and for every partition's manifest to empty (cutover), then consumes from offset 0 and verifies all records across all partitions, contiguous per partition. Exercises the per-partition cutover path and the sink's ntp -> topic_id_partition resolution under fan-out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A graceful broker restart while a partition is mid-migration (the common operational case, distinct from total-loss cluster recovery). The migration state -- the archival STM manifest and the migration flag -- lives in the partition raft log, so it must survive the restart: the partition comes back still migrating (manifest non-empty), the mirror resumes, and it cuts over. No wipe, so every acked record is still served. A lost flag/manifest would treat the partition as a fresh cloud topic and drop the tiered data, which the consume-all (offset_gaps=0, valid_reads>=acked) catches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A read replica created against a migrating source and kept live across the source's migrating->complete flip. The replica is created while the source is still migrating (serving imported extents); the source then finishes and cuts over to a native cloud topic. The replica's snapshot refreshes to the completed source (its high watermark catches up to the full record count) and a consumer from offset 0 sees every source record, contiguous. Validates E2's complete branch and the phase flip on the replica (the existing RR e2e only covers the migrating branch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Iceberg translation must stay continuous across a cutover-last TS->CT migration: during migration the partition is served as tiered storage and translation reads it there; at cutover it becomes a native cloud topic (imported extents + native residual in L1) and translation reads it there. Translation resumes from its persisted highest_translated_offset (carried in the partition raft log across the transition), so the Iceberg table must cover the full offset range with no gap or duplicate. The test produces phase-1 (TS) + phase-2 (across the trigger), waits for cutover, waits for translation to reach the high watermark, and uses DatalakeVerifier to cross-check every offset against the Iceberg table. Matrixed over storage mode (Trino query engine). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Whole-cluster recovery of a partition whose tiered->cloud migration has completed: it must recover as a cloud topic served from L1, not be routed back to tiered storage by the stale remote archival manifest. Deterministic: waits for the L1 metastore manifest to be uploaded (so recovery restores the full L1) while leaving the remote archival manifest stale, then asserts a from-0 consume reaches the high watermark with every record. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two cases covering the storage-mode migration trigger semantics. trigger_feature_gated: the tiered -> cloud trigger is rejected while the tiered_to_cloud_migration cluster feature is inactive and accepted once it is active (the partition then migrates and cuts over) -- the F1 mixed-version gate, so a migration does not begin while older nodes that cannot drive it remain in the cluster. retrigger_idempotent: re-issuing the trigger before cutover is a no-op (no error, no second migration) and the partition still cuts over exactly once with nothing lost; a cut-over cloud topic cannot be reverted to tiered storage (cloud -> tiered is rejected -- migration is one-way); post-cutover writes still reconcile into L1 with the archival manifest staying empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A graceful broker restart immediately after cutover must bring the partition back as a fully-cut-over cloud topic: the archival manifest stays empty (it must not revert to tiered storage by rebuilding from the still-present remote manifest), every record is still served from L1, and post-restart writes reconcile into L1 (the cutover baseline survived). The graceful-restart analogue of the completed-migration recovery e2e -- restart replays the raft log rather than recovering from cloud metadata. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
End-to-end migration of a replicated (RF=3) partition: the archival STM and the cutover batch (reset_metadata) replicate through a real raft quorum rather than the single-replica path the other cloud_topics cases use. The partition must converge, cut over, and serve every record from offset 0. Introduces TsToCtMigrationReplicatedTest, a num_brokers=3 harness (its own helpers; deliberately not a subclass of the single-broker base class so that class's tests are not re-run under RF=3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The migration mirror runs only on the partition leader, so a leadership change while migrating must hand off cleanly. Held in the migrating phase via the test knob so the transfer lands mid-mirror: after moving leadership to the next replica the migration state must survive (still TS-served, not reset to a fresh cloud topic), and once the hold is released the new leader's mirror resumes and cuts over with nothing lost. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ry e2e test_ts_to_ct_migration_delete_records: a DeleteRecords (prefix truncation) issued while the partition is migrating head-prunes the imported L1 extents, and the trimmed start survives cutover (deleted records are not resurrected from the imported extent's base offset). test_ts_to_ct_migration_partition_api_battery: exercises the replicated_partition Kafka-facing API surface while the partition is held mid-migration -- list-offsets (earliest + latest), fetch from earliest and from a mid offset, timequery, and a read_committed scan -- then cuts over and re-reads from L1. This is the routing / storage_mode-vs-manifest corner the DeleteRecords and timequery bugs lived in; the battery is what surfaced the timequery-mid-migration defect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A consumer group consumes the pre-migration (tiered) data and commits; the source then cuts over to a cloud topic and more is produced, and the same group resumes from its committed offset -- reading the new records without re-reading from 0 and without skipping. The cutover-last migration preserves the kafka offset space (imported extents share the source offsets), so the committed offset still points at the right record across the TS->CT boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-migration log spans multiple raft terms (forced via leadership transfers while producing on an RF=3 partition), so the archival manifest and the imported L1 extents after cutover carry segments produced under several leader epochs. A consumer from offset 0 must see monotonic, per-term-correct leader epochs across the cut-over partition (kgo-verifier asserts leader-epoch monotonicity), confirming the mirror imports the epochs faithfully. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forces spillover (segment-based; the size threshold disabled) so the pre-migration log spans several spillover sub-manifests, produces all-committed transactions, migrates and cuts over, then checks the whole range survived: the partition starts at offset 0 (the spilled prefix was imported, not pruned) with the high watermark covering everything produced, and a read_committed random spot-check across the range validates content. A full sequential read-back is impractically slow -- the imported region is served by reading the underlying TS segment objects on demand -- so it samples random offsets instead. Regresses the mirror's spillover-archive import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two offline-consume-path edge cases for the migration: recovery_trimmed_head: whole-cluster recovery of a mid-migration partition whose head was trimmed by a DeleteRecords while migrating. After recovery the partition comes back at the trimmed start (not reset to 0, no gap), and the prefix is readable from there. Trims near the tail (DeleteRecords is deterministic, unlike retention timing) so the retained range -- read from tiered storage on demand -- stays small. read_replica_transactions: read_committed on a read replica of a migrating source. The source produces aborted transactions; the replica reads the source's imported extents via L1, and a read_committed consumer on the replica sees only committed records -- the snapshot imported-reader strips aborted ranges independently of the source's _rm_stm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two variants of triggering the tiered->cloud migration on a partition
whose archival manifest is empty -- the structural routing gate then
sends it straight to the cloud-topic path, never through
replicated_partition, and the mirror skips it (no import, no cutover):
* local_data: data produced but not yet uploaded (one large open
segment that never seals) at the trigger. Asserts the local-only
records are not stranded by the routing flip -- contiguous consume
from 0 and reconciliation into L1, not just survival in the local
log.
* empty_topic: trigger on a never-written topic, then produce; behaves
as a native cloud topic (served from 0, manifest stays empty,
reconciled into L1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- test_read_replica_storage_mode_alter_rejected: a read replica of a
plain tiered source must reject tiered->cloud (and tiered->local)
storage-mode changes -- both otherwise-permitted transitions, so the
rejection isolates the read-replica guard.
- test_ts_to_ct_migration_compacted_reupload_mid_migration: holds a
compacted topic migrating and overwrites a large key set to force
recompaction of already-mirrored segments. The mirror must import
cleanly (no inverted-extent stall from a kafka-empty segment), every
imported extent must reference a live object (ValidatePartition
object-storage check), and the partition must serve through cutover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FOLD INTO 36aba4e (tests: add head-prune-during-migration e2e). The post-cutover `trimmed_start > 0` check asserted once, immediately after _wait_for_cutover -- which only waits for the archival manifest to empty. The cloud-topic (L1) pruned start offset can take a moment longer to surface on the describe/list-offsets path, so the single read occasionally caught a transient 0 (observed as a flake in the tiered_cloud variant). Wrap it in a wait_until so the test tolerates that propagation window; a genuinely lost start still fails after the timeout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sjust-redpanda
force-pushed
the
sjust/ts-ct-migration-pr8
branch
from
b87f064 to
6f1f3c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 NOT YET READY FOR REVIEW
Summary
Enable the tiered→cloud migration trigger behind a feature flag, and add the end-to-end ducktape suite that exercises the whole stack. This is the PR that makes a partition actually able to migrate; everything below it is inert without it.
Stacked on PR5 + PR6 + PR7. Base is
dev, so the diff is the entire stack (PR1–PR8); the commits new in this PR are the feature-gate and thetests:ducktape e2e commits.What's in this PR
features: gate the tiered→cloud migration trigger behind a feature flag — until enabled, no partition can enter the migrating state, so the whole stack is behavior-preserving.tests/rptest/tests/cloud_topics/ts_to_ct_migration_test.py): cutover transition, head-prune, multi-partition, node-restart, transactions, DeleteRecords, consumer-group offset continuity, multi-term mirror convergence, transactional completion over a spilled manifest, recovery of a trimmed head, read-replica (incl. across cutover and of a mid-migration source), compaction-across-migration, Iceberg continuity, cluster recovery (mid-migration and completed), feature-gate + idempotency, leadership transfer during mirror, RF=3, liveness under concurrent writes, no-archived-segments, and compacted-reupload. (The DNM live-through-cutover read-replica future-work case is intentionally not part of this PR — it lives only on the integration branch.)Testing
🤖 Generated with Claude Code
Design Doc
PR 0 · PR 1 · PR 2 · PR 3 · PR 4 · PR 5 · PR 6 · PR 7 · PR 8