Skip to content

Commit 3436492

Browse files
mfernestclaudeFeediver1github-actions[bot]
authored
fix(DOC-1919): address review feedback on aws-iam-aurora guide (#396)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Joyce Fee <joyce@redpanda.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 1549fa6 commit 3436492

2 files changed

Lines changed: 82 additions & 55 deletions

File tree

docs-data/connect-diff-4.94.0_to_4.94.1.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"comparison": {
33
"oldVersion": "4.94.0",
44
"newVersion": "4.94.1",
5-
"timestamp": "2026-06-01T12:12:48.726Z"
5+
"timestamp": "2026-06-02T19:59:10.576Z"
66
},
77
"summary": {
88
"newComponents": 0,
Lines changed: 81 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,62 @@
11
= Authenticate to Amazon Aurora using IAM roles
22
// tag::single-source[]
33
:description: Configure IAM roles to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials.
4+
:page-topic-type: how-to
5+
:personas: streaming_developer, platform_operator
6+
:learning-objective-1: Create an IAM database connect role with the required permission policy and trust relationship
7+
:learning-objective-2: Grant the Redpanda Connect pipeline role cross-account access to Aurora
8+
:learning-objective-3: Configure a Redpanda Connect pipeline input to authenticate to Aurora using IAM roles
49

5-
On AWS, you can use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora without storing static credentials.
10+
Use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without storing static credentials.
611

7-
== How it works
12+
Authentication uses a two-hop role chain. The Redpanda Connect pod assumes a pipeline IAM role through IRSA. That role then assumes a database-specific role in your Aurora account, which generates a short-lived RDS IAM token that Aurora accepts as a password.
813

9-
Authentication happens through a two-hop role chain:
14+
This page covers configuring both roles with the required trust relationships. Redpanda manages the pipeline role and you add a single inline policy to it. You create and own the database connect role.
1015

11-
. The Redpanda Connect pod uses a projected OIDC token to assume the cluster's pipeline IAM role (`redpanda-<CLUSTER_ID>-redpanda-connect-pipeline`) using `sts:AssumeRoleWithWebIdentity` (IRSA).
12-
. The pipeline role assumes a database-specific role in your Aurora account using `sts:AssumeRole`.
13-
. The database role generates a short-lived RDS IAM authentication token that Aurora accepts as a password.
16+
After reading this page, you will be able to:
1417

15-
Both roles must exist with trust relationships that permit this chain. The pipeline role is managed by Redpanda — you add a single inline policy to it. The database connect role is created and owned by you.
18+
* [ ] {learning-objective-1}
19+
* [ ] {learning-objective-2}
20+
* [ ] {learning-objective-3}
1621
22+
[[prerequisites]]
1723
== Prerequisites
1824

1925
* Redpanda Connect deployed on AWS
2026
* An Aurora cluster (PostgreSQL or MySQL)
2127
* Permissions to create IAM roles and attach inline policies in both the Redpanda and Aurora AWS accounts
22-
* An IAM role associated with the Redpanda Connect pod (see <<Step 1: Find the pipeline IAM role name>>)
28+
* An IAM role associated with the Redpanda Connect pod (see <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>)
2329

24-
== Step 1: Find the pipeline IAM role name
30+
[[find-pipeline-iam-role-name]]
31+
== Find the pipeline IAM role name
2532

2633
For Redpanda Cloud BYOC deployments, the pipeline role is pre-created in the AWS account where your cluster is deployed and follows this naming convention:
2734

2835
----
29-
redpanda-<CLUSTER_ID>-redpanda-connect-pipeline
36+
redpanda-<cluster-id>-redpanda-connect-pipeline
3037
----
3138

32-
CAUTION: Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. Only add new inline policies.
39+
CAUTION: Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. To avoid conflicts, add only new inline policies to this role.
3340

3441
For self-managed deployments, use the IAM role associated with your Redpanda Connect pod or EC2 instance.
3542

36-
== Step 2: Find the Aurora cluster resource ID
43+
[[find-aurora-cluster-resource-id]]
44+
== Find the Aurora cluster resource ID
3745

38-
The IAM policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Retrieve it with:
46+
The policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Run the following command to retrieve the resource ID:
3947

4048
[source,bash]
4149
----
4250
aws rds describe-db-clusters \
4351
--query "DBClusters[?DBClusterIdentifier=='<cluster-name>'].[DbClusterResourceId]" \
4452
--output text \
45-
--profile <db_account_aws_profile>
53+
--profile <db-account-aws-profile>
4654
----
4755

4856
The resource ID has the format `cluster-<alphanumeric-string>`.
4957

50-
== Step 3: Create the database connect role (Aurora account)
58+
[[create-database-connect-role]]
59+
== Create the database connect role
5160

5261
In the AWS account where Aurora is hosted, create an IAM role with the following permission policy:
5362

@@ -60,19 +69,20 @@ In the AWS account where Aurora is hosted, create an IAM role with the following
6069
{
6170
"Action": "rds-db:connect",
6271
"Effect": "Allow",
63-
"Resource": "arn:aws:rds-db:<REGION>:<DB_ACCT_ID>:dbuser:cluster-<RESOURCE_ID>/<iam_db_user>"
72+
"Resource": "arn:aws:rds-db:<region>:<db-account-id>:dbuser:cluster-<resource-id>/<iam-db-user>"
6473
}
6574
]
6675
}
6776
----
6877

6978
Replace the following placeholders:
7079

71-
* `<REGION>`: The AWS region of the Aurora cluster
72-
* `<DB_ACCT_ID>`: The AWS account ID that hosts Aurora
73-
* `<RESOURCE_ID>`: The cluster resource ID from <<Step 2: Find the Aurora cluster resource ID>>
74-
* `<iam_db_user>`: The database user configured for IAM authentication
80+
* `<region>`: The AWS region of the Aurora cluster
81+
* `<db-account-id>`: The AWS account ID that hosts Aurora
82+
* `<resource-id>`: The cluster resource ID from <<find-aurora-cluster-resource-id,Find the Aurora cluster resource ID>>
83+
* `<iam-db-user>`: The database user configured for IAM authentication
7584

85+
[[trust-policy]]
7686
=== Trust policy
7787

7888
Attach the following trust policy to the role to allow the Redpanda pipeline role to assume it:
@@ -86,7 +96,7 @@ Attach the following trust policy to the role to allow the Redpanda pipeline rol
8696
{
8797
"Effect": "Allow",
8898
"Principal": {
89-
"AWS": "arn:aws:iam::<RP_ACCT_ID>:role/<PIPELINE_ROLE_NAME>"
99+
"AWS": "arn:aws:iam::<rp-account-id>:role/<pipeline-role-name>"
90100
},
91101
"Action": "sts:AssumeRole"
92102
}
@@ -96,11 +106,12 @@ Attach the following trust policy to the role to allow the Redpanda pipeline rol
96106

97107
Replace the following placeholders:
98108

99-
* `<RP_ACCT_ID>`: The AWS account ID where the Redpanda Connect pipeline role lives
100-
* `<PIPELINE_ROLE_NAME>`: The pipeline role name from <<Step 1: Find the pipeline IAM role name>>
109+
* `<rp-account-id>`: The AWS account ID where the Redpanda Connect pipeline role lives
110+
* `<pipeline-role-name>`: The pipeline role name from <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>
101111

102112
NOTE: The database connect role must be owned by the same AWS account as the Aurora instance. For cross-account setups, create this role in the Aurora account, not the Redpanda account.
103113

114+
[[required-tag]]
104115
=== Required tag
105116

106117
The database connect role must have the following tag. Without it, the pipeline role cannot assume the database role.
@@ -111,32 +122,34 @@ The database connect role must have the following tag. Without it, the pipeline
111122
|`redpanda_scope_redpanda_connect` |`true`
112123
|===
113124

125+
[[create-the-role]]
114126
=== Create the role
115127

116128
Run the following AWS CLI commands to create the role and apply the permission policy and tag:
117129

118130
[source,bash]
119131
----
120132
aws iam create-role \
121-
--role-name "<DB_CONNECT_ROLE_NAME>" \
133+
--role-name "<db-connect-role-name>" \
122134
--assume-role-policy-document file://trust-policy.json \
123-
--profile <db_account_aws_profile>
135+
--profile <db-account-aws-profile>
124136
125137
aws iam put-role-policy \
126-
--role-name "<DB_CONNECT_ROLE_NAME>" \
138+
--role-name "<db-connect-role-name>" \
127139
--policy-name "aurora-rds-connect" \
128140
--policy-document file://permission-policy.json \
129-
--profile <db_account_aws_profile>
141+
--profile <db-account-aws-profile>
130142
131143
aws iam tag-role \
132-
--role-name "<DB_CONNECT_ROLE_NAME>" \
144+
--role-name "<db-connect-role-name>" \
133145
--tags Key=redpanda_scope_redpanda_connect,Value=true \
134-
--profile <db_account_aws_profile>
146+
--profile <db-account-aws-profile>
135147
----
136148

137-
== Step 4: Grant the pipeline role cross-account access (Redpanda account)
149+
[[grant-pipeline-cross-account-access]]
150+
== Grant the pipeline role cross-account access
138151

139-
In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in <<Step 1: Find the pipeline IAM role name>>) to allow it to assume your database connect role:
152+
In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>) to allow it to assume your database connect role:
140153

141154
.inline-policy.json
142155
[source,json]
@@ -147,7 +160,7 @@ In the AWS account where the Redpanda Connect pipeline role lives, add an inline
147160
{
148161
"Effect": "Allow",
149162
"Action": "sts:AssumeRole",
150-
"Resource": "arn:aws:iam::<DB_ACCT_ID>:role/<DB_CONNECT_ROLE_NAME>",
163+
"Resource": "arn:aws:iam::<db-account-id>:role/<db-connect-role-name>",
151164
"Condition": {
152165
"StringEquals": {
153166
"aws:ResourceTag/redpanda_scope_redpanda_connect": "true"
@@ -158,28 +171,29 @@ In the AWS account where the Redpanda Connect pipeline role lives, add an inline
158171
}
159172
----
160173

161-
Attach the policy using the AWS CLI:
174+
To attach the policy, run:
162175

163176
[source,bash]
164177
----
165178
aws iam put-role-policy \
166-
--role-name "<PIPELINE_ROLE_NAME>" \
179+
--role-name "<pipeline-role-name>" \
167180
--policy-name "allow-x-account-db-connect" \
168181
--policy-document file://inline-policy.json \
169-
--profile <redpanda_aws_profile>
182+
--profile <redpanda-aws-profile>
170183
----
171184

172-
To verify the policy is attached:
185+
To verify the policy is attached, run:
173186

174187
[source,bash]
175188
----
176189
aws iam get-role-policy \
177-
--role-name "<PIPELINE_ROLE_NAME>" \
190+
--role-name "<pipeline-role-name>" \
178191
--policy-name "allow-x-account-db-connect" \
179-
--profile <redpanda_aws_profile>
192+
--profile <redpanda-aws-profile>
180193
----
181194

182-
== Step 5: Configure the network
195+
[[configure-the-network]]
196+
== Configure the network
183197

184198
The Aurora security group must allow inbound traffic from your Redpanda Connect cluster's outbound IP addresses on the database port (5432 for PostgreSQL, 3306 for MySQL).
185199

@@ -193,15 +207,18 @@ AUTH_TOKEN=$(curl -s --request POST \
193207
--data grant_type=client_credentials \
194208
--data client_id="${REDPANDA_CLIENT_ID}" \
195209
--data client_secret="${REDPANDA_CLIENT_SECRET}" \
196-
--data audience=cloudv2-production.redpanda.cloud | jq -r '.access_token')
210+
--data audience=cloudv2-production.redpanda.cloud \
211+
| jq -r '.access_token')
197212
198213
curl -s -X GET "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \
199-
-H "Authorization: Bearer ${AUTH_TOKEN}" | jq .cluster.nat_gateways
214+
-H "Authorization: Bearer ${AUTH_TOKEN}" \
215+
| jq .cluster.nat_gateways
200216
----
201217

202-
== Step 6: Configure the pipeline
218+
[[configure-the-pipeline]]
219+
== Configure the pipeline
203220

204-
In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in <<Step 3: Create the database connect role (Aurora account)>>.
221+
In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in <<create-database-connect-role,Create the database connect role>>.
205222

206223
This configuration applies to the `postgres_cdc`, `pg_stream`, and `mysql_cdc` inputs.
207224

@@ -211,13 +228,13 @@ For the `postgres_cdc` and `pg_stream` inputs:
211228
----
212229
input:
213230
postgres_cdc:
214-
dsn: "host=<aurora-endpoint> port=5432 user=<iam_db_user> dbname=<db_name> sslmode=require"
231+
dsn: "host=<aurora-endpoint> port=5432 user=<iam-db-user> dbname=<db-name> sslmode=require"
215232
aws:
216233
enabled: true
217-
region: <REGION>
234+
region: <region>
218235
endpoint: <aurora-endpoint>
219236
roles:
220-
- role: arn:aws:iam::<DB_ACCT_ID>:role/<DB_CONNECT_ROLE_NAME>
237+
- role: arn:aws:iam::<db-account-id>:role/<db-connect-role-name>
221238
----
222239

223240
For the `mysql_cdc` input:
@@ -226,33 +243,43 @@ For the `mysql_cdc` input:
226243
----
227244
input:
228245
mysql_cdc:
229-
dsn: "<iam_db_user>@tcp(<aurora-endpoint>:3306)/<db_name>?tls=true"
246+
dsn: "<iam-db-user>@tcp(<aurora-endpoint>:3306)/<db-name>?tls=true"
230247
aws:
231248
enabled: true
232-
region: <REGION>
249+
region: <region>
233250
endpoint: <aurora-endpoint>
234251
roles:
235-
- role: arn:aws:iam::<DB_ACCT_ID>:role/<DB_CONNECT_ROLE_NAME>
252+
- role: arn:aws:iam::<db-account-id>:role/<db-connect-role-name>
236253
----
237254

238255
The `aws.roles` field accepts an array of role ARNs, which are assumed in sequence. This supports chaining multiple role assumptions for more complex cross-account setups.
239256

240-
== Troubleshoot
257+
[[troubleshooting]]
258+
== Troubleshooting
241259

242-
=== is not authorized to perform: sts:AssumeRole
260+
These errors appear in Redpanda Connect pipeline logs. Monitor logs using `+rpk topic consume __redpanda.connect.logs+`.
261+
262+
[[err-sts-assumerole]]
263+
=== `is not authorized to perform: sts:AssumeRole`
243264

244265
The inline assume-role policy is missing from the Redpanda Connect pipeline role, or the tag condition is not met.
245266

246267
. Verify that the database connect role has the `redpanda_scope_redpanda_connect: true` tag.
247-
. Attach the inline policy as described in <<Step 4: Grant the pipeline role cross-account access (Redpanda account)>>.
268+
. Attach the inline policy as described in <<grant-pipeline-cross-account-access,Grant the pipeline role cross-account access>>.
248269

249-
The error clears automatically once the policy is in place. Monitor pipeline logs to confirm:
270+
The error clears automatically after the policy is in place. Monitor pipeline logs to confirm:
250271

251272
[source,bash]
252273
----
253-
rpk topic consume __redpanda.connect.logs --offset end | grep '<PIPELINE_ID>'
274+
rpk topic consume __redpanda.connect.logs --offset end | grep '<pipeline-id>'
254275
----
255276

256277
Look for a message like `postgres_cdc input go active` or `mysql_cdc input go active` to confirm successful authentication.
257278

279+
[[next-steps]]
280+
== Next steps
281+
282+
* xref:components:inputs/postgres_cdc.adoc[`postgres_cdc` input]
283+
* xref:components:inputs/mysql_cdc.adoc[`mysql_cdc` input]
284+
258285
// end::single-source[]

0 commit comments

Comments
 (0)