Skip to content

Commit 694e7b5

Browse files
twmbclaudeFeediver1
authored
docs: document required ACLs for the Redpanda Migrator user (#451)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
1 parent 5ef69c0 commit 694e7b5

3 files changed

Lines changed: 109 additions & 0 deletions

File tree

modules/components/pages/inputs/redpanda_migrator.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ For capabilities, guarantees, scheduling, and examples, see the output documenta
5151
* Must be paired with a `redpanda_migrator` output in the same pipeline.
5252
* Requires access to a source Kafka or Redpanda cluster.
5353
* Consumer group configuration is recommended for partition balancing.
54+
* When the source cluster enforces ACLs, a consumer ACL alone is not enough for the source principal: it needs at minimum topic `READ` *and* `DESCRIBE_CONFIGS`, plus consumer group and cluster permissions. A `READ` ACL grants `DESCRIBE` but not `DESCRIBE_CONFIGS`, so the migrator consumes messages but fails to create topics with `TOPIC_AUTHORIZATION_FAILED`. See xref:cookbooks:redpanda_migrator.adoc#required-permissions[Required permissions].
5455

5556
== Multiple migrator pairs
5657
When using multiple migrator pairs in a single pipeline, coordination is based on the `label` field. The label of the input and output must match exactly for correct pairing. If labels do not match, migration fails for that pair.

modules/components/pages/outputs/redpanda_migrator.adoc

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,16 @@ include::components:example$advanced/outputs/redpanda_migrator.yaml[]
4242
======
4343

4444

45+
== Requirements
46+
47+
When the destination cluster enforces ACLs, the destination principal needs permission to create topics and add partitions, not only to produce records. Grant these ACLs at minimum:
48+
49+
* Topic `CREATE`, `WRITE`, `ALTER`, and `DESCRIBE_CONFIGS`. Cluster `CREATE` also authorizes topic creation.
50+
* When consumer group migration is enabled, consumer group `READ`.
51+
* When `sync_topic_acls` is enabled, cluster `ALTER`.
52+
53+
For the source-principal ACLs and full details, see xref:cookbooks:redpanda_migrator.adoc#required-permissions[Required permissions].
54+
4555
== Multiple migrator pairs
4656

4757
Each migrator pair requires a unique `label`. Set the same label value on both the input and output within a pair. Labels must match exactly; mismatched labels prevent the input and output from coordinating.

modules/cookbooks/pages/redpanda_migrator.adoc

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,104 @@ endif::[]
365365

366366
At this point, the `source` cluster has some data in both `foo` and `bar` topics, and the consumer prints the messages it reads from these topics to `stdout`.
367367

368+
[#required-permissions]
369+
== Required permissions
370+
371+
This cookbook authenticates as a superuser for simplicity. In production, when the source and destination clusters enforce authorization (ACLs), a basic consumer or producer ACL is not sufficient for Redpanda Migrator. The migrator authenticates to the source cluster with the credentials in the `redpanda_migrator` input, and to the destination cluster with the credentials in the `redpanda_migrator` output, so grant ACLs to each principal as follows.
372+
373+
[IMPORTANT]
374+
====
375+
To recreate each topic on the destination with matching settings, Redpanda Migrator reads the source topic's configuration with a `DescribeConfigs` request, which requires the `DESCRIBE_CONFIGS` operation on the topic.
376+
377+
A consumer ACL (`READ`) implicitly grants `DESCRIBE`, but it does *not* grant `DESCRIBE_CONFIGS`. If the source principal has only `READ`, the migrator consumes messages successfully but fails to create topics, logging an error such as:
378+
379+
[,text]
380+
----
381+
level=error msg="Failed to send message to redpanda_migrator: creating records: sync topics:
382+
create topic <topic>: get topic details <topic>: TOPIC_AUTHORIZATION_FAILED:
383+
Not authorized to access topics: [Topic authorization failed.]"
384+
----
385+
386+
Despite the `create topic` wording, this failure is the `DescribeConfigs` read against the *source* cluster, not the topic creation on the destination.
387+
====
388+
389+
=== Source cluster
390+
391+
Grant the source principal (the `redpanda_migrator` input credentials) these ACLs:
392+
393+
[cols="1,1,2"]
394+
|===
395+
| Resource | Operations | Purpose
396+
397+
| Topic (migrated topics)
398+
| `READ`, `DESCRIBE_CONFIGS`
399+
| Consume records (`READ`, which also grants `DESCRIBE` for metadata and offsets) and read topic configurations to replicate them (`DESCRIBE_CONFIGS`).
400+
401+
| Group (the input's `consumer_group`)
402+
| `READ`
403+
| Join the migrator's own consumer group and track progress.
404+
405+
| Group (migrated groups)
406+
| `DESCRIBE`
407+
| Read source consumer group offsets. Required when consumer group migration is enabled (`consumer_groups.enabled`, the default).
408+
409+
| Cluster
410+
| `DESCRIBE`
411+
| List source consumer groups. Also required to read source ACLs when `sync_topic_acls` is enabled on the output.
412+
|===
413+
414+
For example, to grant the least-privilege source ACLs to `User:migrator` with `rpk`:
415+
416+
[,bash]
417+
----
418+
# Consume records and read topic configs (READ also grants DESCRIBE; DESCRIBE_CONFIGS does not come with READ)
419+
rpk security acl create --allow-principal User:migrator \
420+
--operation read,describe_configs \
421+
--topic <migrated-topics>
422+
423+
# READ on the migrator's own consumer group
424+
rpk security acl create --allow-principal User:migrator \
425+
--operation read \
426+
--group <migrator-consumer-group>
427+
428+
# DESCRIBE on the groups being migrated (omit if consumer_groups.enabled is false)
429+
rpk security acl create --allow-principal User:migrator \
430+
--operation describe \
431+
--group <migrated-groups>
432+
433+
# List consumer groups (and describe ACLs if sync_topic_acls is enabled)
434+
rpk security acl create --allow-principal User:migrator \
435+
--operation describe \
436+
--cluster
437+
----
438+
439+
=== Destination cluster
440+
441+
Grant the destination principal (the `redpanda_migrator` output credentials) these ACLs:
442+
443+
[cols="1,1,2"]
444+
|===
445+
| Resource | Operations | Purpose
446+
447+
| Topic (migrated topics)
448+
| `CREATE`, `WRITE`, `ALTER`, `DESCRIBE_CONFIGS`
449+
| Create topics (`CREATE` and `DESCRIBE_CONFIGS`), produce migrated records (`WRITE`), and add partitions to match the source (`ALTER`). These operations also grant `DESCRIBE`.
450+
451+
| Cluster
452+
| `CREATE`
453+
| Allow creating destination topics whose names are not known in advance. Use instead of per-topic `CREATE`.
454+
455+
| Group (migrated groups)
456+
| `READ`
457+
| Commit translated consumer group offsets. Required when consumer group migration is enabled (`consumer_groups.enabled`, the default).
458+
459+
| Cluster
460+
| `ALTER`
461+
| Create migrated ACLs on the destination. Required only when `sync_topic_acls` is enabled.
462+
|===
463+
464+
TIP: Run `rpk security acl --help-operations` to see which ACL operation each Kafka request requires.
465+
368466
== Configure and start Redpanda Migrator
369467

370468
The unified Redpanda Migrator does the following:

0 commit comments

Comments
 (0)