You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
:description: Configure IAM roles to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials.
4
+
:page-topic-type: how-to
5
+
:personas: streaming_developer, platform_operator
6
+
:learning-objective-1: Create an IAM database connect role with the required permission policy and trust relationship
7
+
:learning-objective-2: Grant the Redpanda Connect pipeline role cross-account access to Aurora
8
+
:learning-objective-3: Configure a Redpanda Connect pipeline input to authenticate to Aurora using IAM roles
4
9
5
-
On AWS, you can use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora without storing static credentials.
10
+
Use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without storing static credentials.
6
11
7
-
== How it works
12
+
Authentication uses a two-hop role chain. The Redpanda Connect pod assumes a pipeline IAM role through IRSA. That role then assumes a database-specific role in your Aurora account, which generates a short-lived RDS IAM token that Aurora accepts as a password.
8
13
9
-
Authentication happens through a two-hop role chain:
14
+
This page covers configuring both roles with the required trust relationships. Redpanda manages the pipeline role and you add a single inline policy to it. You create and own the database connect role.
10
15
11
-
. The Redpanda Connect pod uses a projected OIDC token to assume the cluster's pipeline IAM role (`redpanda-<CLUSTER_ID>-redpanda-connect-pipeline`) using `sts:AssumeRoleWithWebIdentity` (IRSA).
12
-
. The pipeline role assumes a database-specific role in your Aurora account using `sts:AssumeRole`.
13
-
. The database role generates a short-lived RDS IAM authentication token that Aurora accepts as a password.
16
+
After reading this page, you will be able to:
14
17
15
-
Both roles must exist with trust relationships that permit this chain. The pipeline role is managed by Redpanda — you add a single inline policy to it. The database connect role is created and owned by you.
18
+
* [ ] {learning-objective-1}
19
+
* [ ] {learning-objective-2}
20
+
* [ ] {learning-objective-3}
16
21
22
+
[[prerequisites]]
17
23
== Prerequisites
18
24
19
25
* Redpanda Connect deployed on AWS
20
26
* An Aurora cluster (PostgreSQL or MySQL)
21
27
* Permissions to create IAM roles and attach inline policies in both the Redpanda and Aurora AWS accounts
22
-
* An IAM role associated with the Redpanda Connect pod (see <<Step 1: Find the pipeline IAM role name>>)
28
+
* An IAM role associated with the Redpanda Connect pod (see <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>)
23
29
24
-
== Step 1: Find the pipeline IAM role name
30
+
[[find-pipeline-iam-role-name]]
31
+
== Find the pipeline IAM role name
25
32
26
33
For Redpanda Cloud BYOC deployments, the pipeline role is pre-created in the AWS account where your cluster is deployed and follows this naming convention:
27
34
28
35
----
29
-
redpanda-<CLUSTER_ID>-redpanda-connect-pipeline
36
+
redpanda-<cluster-id>-redpanda-connect-pipeline
30
37
----
31
38
32
-
CAUTION: Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. Only add new inline policies.
39
+
CAUTION: Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. To avoid conflicts, add only new inline policies to this role.
33
40
34
41
For self-managed deployments, use the IAM role associated with your Redpanda Connect pod or EC2 instance.
35
42
36
-
== Step 2: Find the Aurora cluster resource ID
43
+
[[find-aurora-cluster-resource-id]]
44
+
== Find the Aurora cluster resource ID
37
45
38
-
The IAM policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Retrieve it with:
46
+
The policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Run the following command to retrieve the resource ID:
@@ -96,11 +106,12 @@ Attach the following trust policy to the role to allow the Redpanda pipeline rol
96
106
97
107
Replace the following placeholders:
98
108
99
-
* `<RP_ACCT_ID>`: The AWS account ID where the Redpanda Connect pipeline role lives
100
-
* `<PIPELINE_ROLE_NAME>`: The pipeline role name from <<Step 1: Find the pipeline IAM role name>>
109
+
* `<rp-account-id>`: The AWS account ID where the Redpanda Connect pipeline role lives
110
+
* `<pipeline-role-name>`: The pipeline role name from <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>
101
111
102
112
NOTE: The database connect role must be owned by the same AWS account as the Aurora instance. For cross-account setups, create this role in the Aurora account, not the Redpanda account.
103
113
114
+
[[required-tag]]
104
115
=== Required tag
105
116
106
117
The database connect role must have the following tag. Without it, the pipeline role cannot assume the database role.
@@ -111,32 +122,34 @@ The database connect role must have the following tag. Without it, the pipeline
111
122
|`redpanda_scope_redpanda_connect` |`true`
112
123
|===
113
124
125
+
[[create-the-role]]
114
126
=== Create the role
115
127
116
128
Run the following AWS CLI commands to create the role and apply the permission policy and tag:
== Step 4: Grant the pipeline role cross-account access (Redpanda account)
149
+
[[grant-pipeline-cross-account-access]]
150
+
== Grant the pipeline role cross-account access
138
151
139
-
In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in <<Step 1: Find the pipeline IAM role name>>) to allow it to assume your database connect role:
152
+
In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>) to allow it to assume your database connect role:
140
153
141
154
.inline-policy.json
142
155
[source,json]
@@ -147,7 +160,7 @@ In the AWS account where the Redpanda Connect pipeline role lives, add an inline
@@ -158,28 +171,29 @@ In the AWS account where the Redpanda Connect pipeline role lives, add an inline
158
171
}
159
172
----
160
173
161
-
Attach the policy using the AWS CLI:
174
+
To attach the policy, run:
162
175
163
176
[source,bash]
164
177
----
165
178
aws iam put-role-policy \
166
-
--role-name "<PIPELINE_ROLE_NAME>" \
179
+
--role-name "<pipeline-role-name>" \
167
180
--policy-name "allow-x-account-db-connect" \
168
181
--policy-document file://inline-policy.json \
169
-
--profile <redpanda_aws_profile>
182
+
--profile <redpanda-aws-profile>
170
183
----
171
184
172
-
To verify the policy is attached:
185
+
To verify the policy is attached, run:
173
186
174
187
[source,bash]
175
188
----
176
189
aws iam get-role-policy \
177
-
--role-name "<PIPELINE_ROLE_NAME>" \
190
+
--role-name "<pipeline-role-name>" \
178
191
--policy-name "allow-x-account-db-connect" \
179
-
--profile <redpanda_aws_profile>
192
+
--profile <redpanda-aws-profile>
180
193
----
181
194
182
-
== Step 5: Configure the network
195
+
[[configure-the-network]]
196
+
== Configure the network
183
197
184
198
The Aurora security group must allow inbound traffic from your Redpanda Connect cluster's outbound IP addresses on the database port (5432 for PostgreSQL, 3306 for MySQL).
185
199
@@ -193,15 +207,18 @@ AUTH_TOKEN=$(curl -s --request POST \
In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in <<Step 3: Create the database connect role (Aurora account)>>.
221
+
In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in <<create-database-connect-role,Create the database connect role>>.
205
222
206
223
This configuration applies to the `postgres_cdc`, `pg_stream`, and `mysql_cdc` inputs.
207
224
@@ -211,13 +228,13 @@ For the `postgres_cdc` and `pg_stream` inputs:
The `aws.roles` field accepts an array of role ARNs, which are assumed in sequence. This supports chaining multiple role assumptions for more complex cross-account setups.
239
256
240
-
== Troubleshoot
257
+
[[troubleshooting]]
258
+
== Troubleshooting
241
259
242
-
=== is not authorized to perform: sts:AssumeRole
260
+
These errors appear in Redpanda Connect pipeline logs. Monitor logs using `+rpk topic consume __redpanda.connect.logs+`.
261
+
262
+
[[err-sts-assumerole]]
263
+
=== `is not authorized to perform: sts:AssumeRole`
243
264
244
265
The inline assume-role policy is missing from the Redpanda Connect pipeline role, or the tag condition is not met.
245
266
246
267
. Verify that the database connect role has the `redpanda_scope_redpanda_connect: true` tag.
247
-
. Attach the inline policy as described in <<Step 4: Grant the pipeline role cross-account access (Redpanda account)>>.
268
+
. Attach the inline policy as described in <<grant-pipeline-cross-account-access,Grant the pipeline role cross-account access>>.
248
269
249
-
The error clears automatically once the policy is in place. Monitor pipeline logs to confirm:
270
+
The error clears automatically after the policy is in place. Monitor pipeline logs to confirm:
250
271
251
272
[source,bash]
252
273
----
253
-
rpk topic consume __redpanda.connect.logs --offset end | grep '<PIPELINE_ID>'
274
+
rpk topic consume __redpanda.connect.logs --offset end | grep '<pipeline-id>'
254
275
----
255
276
256
277
Look for a message like `postgres_cdc input go active` or `mysql_cdc input go active` to confirm successful authentication.
0 commit comments