You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: modules/components/pages/inputs/redpanda_migrator.adoc
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -51,6 +51,7 @@ For capabilities, guarantees, scheduling, and examples, see the output documenta
51
51
* Must be paired with a `redpanda_migrator` output in the same pipeline.
52
52
* Requires access to a source Kafka or Redpanda cluster.
53
53
* Consumer group configuration is recommended for partition balancing.
54
+
* When the source cluster enforces ACLs, a consumer ACL alone is not enough for the source principal: it needs at minimum topic `READ` *and* `DESCRIBE_CONFIGS`, plus consumer group and cluster permissions. A `READ` ACL grants `DESCRIBE` but not `DESCRIBE_CONFIGS`, so the migrator consumes messages but fails to create topics with `TOPIC_AUTHORIZATION_FAILED`. See xref:cookbooks:redpanda_migrator.adoc#required-permissions[Required permissions].
54
55
55
56
== Multiple migrator pairs
56
57
When using multiple migrator pairs in a single pipeline, coordination is based on the `label` field. The label of the input and output must match exactly for correct pairing. If labels do not match, migration fails for that pair.
When the destination cluster enforces ACLs, the destination principal needs permission to create topics and add partitions, not only to produce records. Grant these ACLs at minimum:
48
+
49
+
* Topic `CREATE`, `WRITE`, `ALTER`, and `DESCRIBE_CONFIGS`. Cluster `CREATE` also authorizes topic creation.
50
+
* When consumer group migration is enabled, consumer group `READ`.
51
+
* When `sync_topic_acls` is enabled, cluster `ALTER`.
52
+
53
+
For the source-principal ACLs and full details, see xref:cookbooks:redpanda_migrator.adoc#required-permissions[Required permissions].
54
+
45
55
== Multiple migrator pairs
46
56
47
57
Each migrator pair requires a unique `label`. Set the same label value on both the input and output within a pair. Labels must match exactly; mismatched labels prevent the input and output from coordinating.
Copy file name to clipboardExpand all lines: modules/cookbooks/pages/redpanda_migrator.adoc
+98Lines changed: 98 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -365,6 +365,104 @@ endif::[]
365
365
366
366
At this point, the `source` cluster has some data in both `foo` and `bar` topics, and the consumer prints the messages it reads from these topics to `stdout`.
367
367
368
+
[#required-permissions]
369
+
== Required permissions
370
+
371
+
This cookbook authenticates as a superuser for simplicity. In production, when the source and destination clusters enforce authorization (ACLs), a basic consumer or producer ACL is not sufficient for Redpanda Migrator. The migrator authenticates to the source cluster with the credentials in the `redpanda_migrator` input, and to the destination cluster with the credentials in the `redpanda_migrator` output, so grant ACLs to each principal as follows.
372
+
373
+
[IMPORTANT]
374
+
====
375
+
To recreate each topic on the destination with matching settings, Redpanda Migrator reads the source topic's configuration with a `DescribeConfigs` request, which requires the `DESCRIBE_CONFIGS` operation on the topic.
376
+
377
+
A consumer ACL (`READ`) implicitly grants `DESCRIBE`, but it does *not* grant `DESCRIBE_CONFIGS`. If the source principal has only `READ`, the migrator consumes messages successfully but fails to create topics, logging an error such as:
378
+
379
+
[,text]
380
+
----
381
+
level=error msg="Failed to send message to redpanda_migrator: creating records: sync topics:
382
+
create topic <topic>: get topic details <topic>: TOPIC_AUTHORIZATION_FAILED:
383
+
Not authorized to access topics: [Topic authorization failed.]"
384
+
----
385
+
386
+
Despite the `create topic` wording, this failure is the `DescribeConfigs` read against the *source* cluster, not the topic creation on the destination.
387
+
====
388
+
389
+
=== Source cluster
390
+
391
+
Grant the source principal (the `redpanda_migrator` input credentials) these ACLs:
392
+
393
+
[cols="1,1,2"]
394
+
|===
395
+
| Resource | Operations | Purpose
396
+
397
+
| Topic (migrated topics)
398
+
| `READ`, `DESCRIBE_CONFIGS`
399
+
| Consume records (`READ`, which also grants `DESCRIBE` for metadata and offsets) and read topic configurations to replicate them (`DESCRIBE_CONFIGS`).
400
+
401
+
| Group (the input's `consumer_group`)
402
+
| `READ`
403
+
| Join the migrator's own consumer group and track progress.
404
+
405
+
| Group (migrated groups)
406
+
| `DESCRIBE`
407
+
| Read source consumer group offsets. Required when consumer group migration is enabled (`consumer_groups.enabled`, the default).
408
+
409
+
| Cluster
410
+
| `DESCRIBE`
411
+
| List source consumer groups. Also required to read source ACLs when `sync_topic_acls` is enabled on the output.
412
+
|===
413
+
414
+
For example, to grant the least-privilege source ACLs to `User:migrator` with `rpk`:
415
+
416
+
[,bash]
417
+
----
418
+
# Consume records and read topic configs (READ also grants DESCRIBE; DESCRIBE_CONFIGS does not come with READ)
Grant the destination principal (the `redpanda_migrator` output credentials) these ACLs:
442
+
443
+
[cols="1,1,2"]
444
+
|===
445
+
| Resource | Operations | Purpose
446
+
447
+
| Topic (migrated topics)
448
+
| `CREATE`, `WRITE`, `ALTER`, `DESCRIBE_CONFIGS`
449
+
| Create topics (`CREATE` and `DESCRIBE_CONFIGS`), produce migrated records (`WRITE`), and add partitions to match the source (`ALTER`). These operations also grant `DESCRIBE`.
450
+
451
+
| Cluster
452
+
| `CREATE`
453
+
| Allow creating destination topics whose names are not known in advance. Use instead of per-topic `CREATE`.
454
+
455
+
| Group (migrated groups)
456
+
| `READ`
457
+
| Commit translated consumer group offsets. Required when consumer group migration is enabled (`consumer_groups.enabled`, the default).
458
+
459
+
| Cluster
460
+
| `ALTER`
461
+
| Create migrated ACLs on the destination. Required only when `sync_topic_acls` is enabled.
462
+
|===
463
+
464
+
TIP: Run `rpk security acl --help-operations` to see which ACL operation each Kafka request requires.
0 commit comments