feat(DOC-2008): document automatic topic creation for redpanda_migrator

docs-data/connect-diff-4.94.0_to_4.94.1.json

@@ -2,7 +2,7 @@
   "comparison": {
     "oldVersion": "4.94.0",
     "newVersion": "4.94.1",
-    "timestamp": "2026-06-02T19:59:10.576Z"
+    "timestamp": "2026-06-04T17:43:09.470Z"
   },
   "summary": {
     "newComponents": 0,

modules/components/pages/outputs/redpanda_migrator.adoc

@@ -6,19 +6,19 @@
 
 component_type_dropdown::[]
 
-A Kafka producer for migrating data between Kafka/Redpanda clusters.
-
-The `redpanda_migrator` output coordinates migration of topics, schemas, and consumer groups from a source Kafka/Redpanda cluster to a destination cluster.
+Migrates topics, schemas, and consumer groups between Kafka and Redpanda clusters.
 
 [IMPORTANT]
 ====
-This output **must** be paired with a xref:components:inputs/redpanda_migrator.adoc[`redpanda_migrator` input] in the same pipeline. Each pipeline requires both input and output components.
+Pair this output with a xref:components:inputs/redpanda_migrator.adoc[`redpanda_migrator` input] in the same pipeline.
 ====
 
 ifndef::env-cloud[]
 Introduced in version 4.37.0.
 endif::[]
 
+The following shows all available configuration fields and their defaults.
+
 [tabs]
 ======
 Common::
@@ -44,104 +44,105 @@ include::components:example$advanced/outputs/redpanda_migrator.yaml[]
 
 == Multiple migrator pairs
 
-When using multiple migrator pairs in a pipeline, match the `label` field exactly between input and output components for correct coordination.
+Each migrator pair requires a unique `label`. Set the same label value on both the input and output within a pair. Labels must match exactly; mismatched labels prevent the input and output from coordinating.
 
 == Performance tuning
 
 For high-throughput workloads, adjust the following settings:
 
-On this output component:
+On this output:
 
-- `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster)
+- `max_in_flight`: Set to the total number of partitions being migrated. Higher values provide no benefit beyond the partition count.
 
-On the paired xref:components:inputs/redpanda_migrator.adoc#performance-tuning[`redpanda_migrator` input component]:
+On the paired xref:components:inputs/redpanda_migrator.adoc#performance-tuning[`redpanda_migrator` input]:
 
-- `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size
-- `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded
+- `partition_buffer_bytes`: Set to `2MB` to increase the per-partition buffer.
+- `max_yield_batch_bytes`: Set to `1MB` to yield larger batches.
 
 == Synchronization details
 
-*Topics*
-
-- Name resolution with interpolation (default: preserve source name)
-- Automatic creation with mirrored partition counts
-- Selectable replication factor (default: inherit from source)
-- Supported topic configuration keys (serverless-aware subset)
-- Optional ACL replication:
-  - Excludes `ALLOW WRITE` entries
-  - Downgrades `ALLOW ALL` to `READ`
-  - Preserves resource pattern type and host filters
-
-*Schema Registry*
-
-- One-shot or periodic syncing
-- Subject selection via include/exclude regex
-- Subject renaming with interpolation
-- Versions: `latest` or `all` (default: `all`)
-- Optional include of soft-deleted subjects
-- ID handling: translate IDs or keep fixed
-- Optional schema normalization
-- Compatibility propagation (per-subject only)
-- Schema metadata/rules not copied in Serverless mode
-
-*Consumer Groups*
-
-- Periodic syncing
-- Group selection using regex
-- Only `Empty` state groups migrated
-- Timestamp-based offset translation (approximate)
-- No rewind guarantee: offsets only move forward
-- Requires matching partition counts
+=== Topics
+
+- Topic names carry over from the source by default. Use the `topic` field with interpolation to rename topics at the destination.
+- The migrator creates each topic at the destination with the same partition count as the source.
+- Replication factor defaults to the source value. Set `topic_replication_factor` to override it.
+- The migrator copies a serverless-aware subset of topic configuration keys.
+- ACL replication is optional (`sync_topic_acls`):
+  - The migrator does not copy `ALLOW WRITE` ACL entries.
+  - The migrator downgrades `ALLOW ALL` ACL entries to `ALLOW READ`.
+  - Resource pattern type and host filters carry over.
+
+=== Schema Registry
+
+- Syncs once at startup, then periodically based on `schema_registry.interval` (default: every 5 minutes). Set `schema_registry.interval: 0s` for a one-time sync only.
+- Use include and exclude regex patterns to filter which subjects are migrated.
+- Use the `subject` field with interpolation to rename subjects at the destination.
+- Schema versions: `latest` (most recent version only) or `all` (full history). Defaults to `all`.
+- Include soft-deleted subjects with `schema_registry.include_deleted`.
+- The migrator can translate schema IDs to new destination IDs or preserve them as-is. See `schema_registry.translate_ids`.
+- Schema normalization is optional (`schema_registry.normalize`).
+- Compatibility settings carry over per-subject. Schema metadata and rules do not carry over in Serverless mode.
+
+=== Consumer groups
+
+- Sync periodically based on `consumer_groups.interval` (default: every 1 minute).
+- Use include and exclude regex patterns to filter which groups are migrated.
+- By default, all groups except those in `Dead` state migrate. Set `consumer_groups.only_empty: true` to migrate only `Empty` state groups.
+- The migrator translates consumer offsets using timestamps. This is approximate and may be imprecise when multiple records share the same timestamp.
+- Offsets only move forward and never rewind at the destination.
+- Source and destination must have matching partition counts for each migrated group.
 
 == How it works
 
-- Topics: Synced on demand. First write triggers creation.
-- Schema Registry: Synced at connect, then as needed.
-- Consumer Groups: Background loop, filtered by topic mappings.
+Each migration component synchronizes on a different schedule:
+
+- *Topics*: Sync from source on startup and every 5 minutes by default, including source topics that have no current data (for example, after retention cleanup). Configure the sync interval with <<sync_topic_interval,`sync_topic_interval`>>, or set it to `0s` to disable periodic sync. When periodic sync is disabled, topics are still created on demand when the first message arrives.
+- *Schema Registry*: Syncs at startup, then periodically as configured.
+- *Consumer groups*: Sync in the background, filtered to the topics being migrated.
 
 == Guarantees
 
-- Topics created with intended partitioning/replication.
-- Existing topics respected. Mismatches logged.
-- Consumer group offsets never rewound.
-- ACL replication excludes unsafe grants.
+The migrator upholds the following guarantees:
+
+- Creates each destination topic with the intended partition count and replication factor.
+- Never overwrites existing destination topics and logs any partition count mismatches.
+- Consumer group offsets never rewind.
+- ACL replication never grants write access at the destination.
 
 == Limitations
 
-- Destination Schema Registry must be in `READWRITE` or `IMPORT` mode.
+- The destination cluster's Schema Registry must be in `READWRITE` or `IMPORT` mode.
 - Offset translation is best-effort.
-- Consumer group migration requires identical partition counts.
+- Consumer group migration requires identical partition counts at source and destination.
 
 == Metrics
 
-The component exposes comprehensive metrics for monitoring migration operations:
-
-[cols="2,1,2,2"]
+[cols="4,1,1,3"]
 |===
 | Metric Name | Type | Labels | Description
 
-|*Topic migration metrics*|||
+|*Topic migration*|||
 |`redpanda_migrator_topics_created_total` | counter |  | Total topics created on destination
 |`redpanda_migrator_topic_create_errors_total` | counter |  | Topic creation errors
 |`redpanda_migrator_topic_create_latency_ns` | timer |  | Topic creation latency (ns)
 
-|*Schema Registry migration metrics*|||
+|*Schema Registry migration*|||
 |`redpanda_migrator_sr_schemas_created_total` | counter |  | Schemas created in destination registry
 |`redpanda_migrator_sr_schema_create_errors_total` | counter |  | Schema creation errors
 |`redpanda_migrator_sr_schema_create_latency_ns` | timer |  | Schema creation latency (ns)
 |`redpanda_migrator_sr_compatibility_updates_total` | counter |  | Compatibility level updates applied
 |`redpanda_migrator_sr_compatibility_update_errors_total` | counter |  | Compatibility update errors
 |`redpanda_migrator_sr_compatibility_update_latency_ns` | timer |  | Compatibility update latency (ns)
 
-|*Consumer group migration metrics*|||
+|*Consumer group migration*|||
 |`redpanda_migrator_cg_offsets_translated_total` | counter | group | Offsets translated per consumer group
 |`redpanda_migrator_cg_offset_translation_errors_total` | counter | group | Offset translation errors per group
 |`redpanda_migrator_cg_offset_translation_latency_ns` | timer | group | Offset translation latency per group (ns)
 |`redpanda_migrator_cg_offsets_committed_total` | counter | group | Offsets committed per consumer group
 |`redpanda_migrator_cg_offset_commit_errors_total` | counter | group | Offset commit errors per group
 |`redpanda_migrator_cg_offset_commit_latency_ns` | timer | group | Offset commit latency per group (ns)
 
-|*Consumer lag metrics*|||
+|*Consumer lag*|||
 |`redpanda_lag` | gauge | topic, partition | Current consumer lag in messages for each topic partition. Shows difference between high water mark and current consumer position.
 |===