Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-data/connect-diff-4.94.0_to_4.94.1.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"comparison": {
"oldVersion": "4.94.0",
"newVersion": "4.94.1",
"timestamp": "2026-06-01T12:12:48.726Z"
"timestamp": "2026-06-02T19:59:10.576Z"
},
"summary": {
"newComponents": 0,
Expand Down
135 changes: 81 additions & 54 deletions modules/guides/pages/cloud/aws-iam-aurora.adoc
Original file line number Diff line number Diff line change
@@ -1,53 +1,62 @@
= Authenticate to Amazon Aurora using IAM roles
// tag::single-source[]
:description: Configure IAM roles to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials.
:page-topic-type: how-to
:personas: streaming_developer, platform_operator
:learning-objective-1: Create an IAM database connect role with the required permission policy and trust relationship
:learning-objective-2: Grant the Redpanda Connect pipeline role cross-account access to Aurora
:learning-objective-3: Configure a Redpanda Connect pipeline input to authenticate to Aurora using IAM roles

On AWS, you can use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora without storing static credentials.
Use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without storing static credentials.

== How it works
Authentication uses a two-hop role chain. The Redpanda Connect pod assumes a pipeline IAM role through IRSA. That role then assumes a database-specific role in your Aurora account, which generates a short-lived RDS IAM token that Aurora accepts as a password.

Authentication happens through a two-hop role chain:
This page covers configuring both roles with the required trust relationships. Redpanda manages the pipeline role and you add a single inline policy to it. You create and own the database connect role.

. The Redpanda Connect pod uses a projected OIDC token to assume the cluster's pipeline IAM role (`redpanda-<CLUSTER_ID>-redpanda-connect-pipeline`) using `sts:AssumeRoleWithWebIdentity` (IRSA).
. The pipeline role assumes a database-specific role in your Aurora account using `sts:AssumeRole`.
. The database role generates a short-lived RDS IAM authentication token that Aurora accepts as a password.
After reading this page, you will be able to:

Both roles must exist with trust relationships that permit this chain. The pipeline role is managed by Redpanda — you add a single inline policy to it. The database connect role is created and owned by you.
* [ ] {learning-objective-1}
* [ ] {learning-objective-2}
* [ ] {learning-objective-3}

[[prerequisites]]
== Prerequisites

* Redpanda Connect deployed on AWS
* An Aurora cluster (PostgreSQL or MySQL)
* Permissions to create IAM roles and attach inline policies in both the Redpanda and Aurora AWS accounts
* An IAM role associated with the Redpanda Connect pod (see <<Step 1: Find the pipeline IAM role name>>)
* An IAM role associated with the Redpanda Connect pod (see <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>)

== Step 1: Find the pipeline IAM role name
[[find-pipeline-iam-role-name]]
== Find the pipeline IAM role name

For Redpanda Cloud BYOC deployments, the pipeline role is pre-created in the AWS account where your cluster is deployed and follows this naming convention:

----
redpanda-<CLUSTER_ID>-redpanda-connect-pipeline
redpanda-<cluster-id>-redpanda-connect-pipeline
----

CAUTION: Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. Only add new inline policies.
CAUTION: Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. To avoid conflicts, add only new inline policies to this role.

For self-managed deployments, use the IAM role associated with your Redpanda Connect pod or EC2 instance.

== Step 2: Find the Aurora cluster resource ID
[[find-aurora-cluster-resource-id]]
== Find the Aurora cluster resource ID

The IAM policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Retrieve it with:
The policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Run the following command to retrieve the resource ID:

[source,bash]
----
aws rds describe-db-clusters \
--query "DBClusters[?DBClusterIdentifier=='<cluster-name>'].[DbClusterResourceId]" \
--output text \
--profile <db_account_aws_profile>
--profile <db-account-aws-profile>
----

The resource ID has the format `cluster-<alphanumeric-string>`.

== Step 3: Create the database connect role (Aurora account)
[[create-database-connect-role]]
== Create the database connect role

In the AWS account where Aurora is hosted, create an IAM role with the following permission policy:

Expand All @@ -60,19 +69,20 @@ In the AWS account where Aurora is hosted, create an IAM role with the following
{
"Action": "rds-db:connect",
"Effect": "Allow",
"Resource": "arn:aws:rds-db:<REGION>:<DB_ACCT_ID>:dbuser:cluster-<RESOURCE_ID>/<iam_db_user>"
"Resource": "arn:aws:rds-db:<region>:<db-account-id>:dbuser:cluster-<resource-id>/<iam-db-user>"
}
]
}
----

Replace the following placeholders:

* `<REGION>`: The AWS region of the Aurora cluster
* `<DB_ACCT_ID>`: The AWS account ID that hosts Aurora
* `<RESOURCE_ID>`: The cluster resource ID from <<Step 2: Find the Aurora cluster resource ID>>
* `<iam_db_user>`: The database user configured for IAM authentication
* `<region>`: The AWS region of the Aurora cluster
* `<db-account-id>`: The AWS account ID that hosts Aurora
* `<resource-id>`: The cluster resource ID from <<find-aurora-cluster-resource-id,Find the Aurora cluster resource ID>>
* `<iam-db-user>`: The database user configured for IAM authentication

[[trust-policy]]
=== Trust policy

Attach the following trust policy to the role to allow the Redpanda pipeline role to assume it:
Expand All @@ -86,7 +96,7 @@ Attach the following trust policy to the role to allow the Redpanda pipeline rol
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<RP_ACCT_ID>:role/<PIPELINE_ROLE_NAME>"
"AWS": "arn:aws:iam::<rp-account-id>:role/<pipeline-role-name>"
},
"Action": "sts:AssumeRole"
}
Expand All @@ -96,11 +106,12 @@ Attach the following trust policy to the role to allow the Redpanda pipeline rol

Replace the following placeholders:

* `<RP_ACCT_ID>`: The AWS account ID where the Redpanda Connect pipeline role lives
* `<PIPELINE_ROLE_NAME>`: The pipeline role name from <<Step 1: Find the pipeline IAM role name>>
* `<rp-account-id>`: The AWS account ID where the Redpanda Connect pipeline role lives
* `<pipeline-role-name>`: The pipeline role name from <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>

NOTE: The database connect role must be owned by the same AWS account as the Aurora instance. For cross-account setups, create this role in the Aurora account, not the Redpanda account.

[[required-tag]]
=== Required tag

The database connect role must have the following tag. Without it, the pipeline role cannot assume the database role.
Expand All @@ -111,32 +122,34 @@ The database connect role must have the following tag. Without it, the pipeline
|`redpanda_scope_redpanda_connect` |`true`
|===

[[create-the-role]]
=== Create the role

Run the following AWS CLI commands to create the role and apply the permission policy and tag:

[source,bash]
----
aws iam create-role \
--role-name "<DB_CONNECT_ROLE_NAME>" \
--role-name "<db-connect-role-name>" \
--assume-role-policy-document file://trust-policy.json \
--profile <db_account_aws_profile>
--profile <db-account-aws-profile>

aws iam put-role-policy \
--role-name "<DB_CONNECT_ROLE_NAME>" \
--role-name "<db-connect-role-name>" \
--policy-name "aurora-rds-connect" \
--policy-document file://permission-policy.json \
--profile <db_account_aws_profile>
--profile <db-account-aws-profile>

aws iam tag-role \
--role-name "<DB_CONNECT_ROLE_NAME>" \
--role-name "<db-connect-role-name>" \
--tags Key=redpanda_scope_redpanda_connect,Value=true \
--profile <db_account_aws_profile>
--profile <db-account-aws-profile>
----

== Step 4: Grant the pipeline role cross-account access (Redpanda account)
[[grant-pipeline-cross-account-access]]
== Grant the pipeline role cross-account access

In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in <<Step 1: Find the pipeline IAM role name>>) to allow it to assume your database connect role:
In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in <<find-pipeline-iam-role-name,Find the pipeline IAM role name>>) to allow it to assume your database connect role:

.inline-policy.json
[source,json]
Expand All @@ -147,7 +160,7 @@ In the AWS account where the Redpanda Connect pipeline role lives, add an inline
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<DB_ACCT_ID>:role/<DB_CONNECT_ROLE_NAME>",
"Resource": "arn:aws:iam::<db-account-id>:role/<db-connect-role-name>",
"Condition": {
"StringEquals": {
"aws:ResourceTag/redpanda_scope_redpanda_connect": "true"
Expand All @@ -158,28 +171,29 @@ In the AWS account where the Redpanda Connect pipeline role lives, add an inline
}
----

Attach the policy using the AWS CLI:
To attach the policy, run:

[source,bash]
----
aws iam put-role-policy \
--role-name "<PIPELINE_ROLE_NAME>" \
--role-name "<pipeline-role-name>" \
--policy-name "allow-x-account-db-connect" \
--policy-document file://inline-policy.json \
--profile <redpanda_aws_profile>
--profile <redpanda-aws-profile>
----

To verify the policy is attached:
To verify the policy is attached, run:

[source,bash]
----
aws iam get-role-policy \
--role-name "<PIPELINE_ROLE_NAME>" \
--role-name "<pipeline-role-name>" \
--policy-name "allow-x-account-db-connect" \
--profile <redpanda_aws_profile>
--profile <redpanda-aws-profile>
----

== Step 5: Configure the network
[[configure-the-network]]
== Configure the network

The Aurora security group must allow inbound traffic from your Redpanda Connect cluster's outbound IP addresses on the database port (5432 for PostgreSQL, 3306 for MySQL).

Expand All @@ -193,15 +207,18 @@ AUTH_TOKEN=$(curl -s --request POST \
--data grant_type=client_credentials \
--data client_id="${REDPANDA_CLIENT_ID}" \
--data client_secret="${REDPANDA_CLIENT_SECRET}" \
--data audience=cloudv2-production.redpanda.cloud | jq -r '.access_token')
--data audience=cloudv2-production.redpanda.cloud \
| jq -r '.access_token')

curl -s -X GET "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \
-H "Authorization: Bearer ${AUTH_TOKEN}" | jq .cluster.nat_gateways
-H "Authorization: Bearer ${AUTH_TOKEN}" \
| jq .cluster.nat_gateways
----

== Step 6: Configure the pipeline
[[configure-the-pipeline]]
== Configure the pipeline

In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in <<Step 3: Create the database connect role (Aurora account)>>.
In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in <<create-database-connect-role,Create the database connect role>>.

This configuration applies to the `postgres_cdc`, `pg_stream`, and `mysql_cdc` inputs.

Expand All @@ -211,13 +228,13 @@ For the `postgres_cdc` and `pg_stream` inputs:
----
input:
postgres_cdc:
dsn: "host=<aurora-endpoint> port=5432 user=<iam_db_user> dbname=<db_name> sslmode=require"
dsn: "host=<aurora-endpoint> port=5432 user=<iam-db-user> dbname=<db-name> sslmode=require"
aws:
enabled: true
region: <REGION>
region: <region>
endpoint: <aurora-endpoint>
roles:
- role: arn:aws:iam::<DB_ACCT_ID>:role/<DB_CONNECT_ROLE_NAME>
- role: arn:aws:iam::<db-account-id>:role/<db-connect-role-name>
----

For the `mysql_cdc` input:
Expand All @@ -226,33 +243,43 @@ For the `mysql_cdc` input:
----
input:
mysql_cdc:
dsn: "<iam_db_user>@tcp(<aurora-endpoint>:3306)/<db_name>?tls=true"
dsn: "<iam-db-user>@tcp(<aurora-endpoint>:3306)/<db-name>?tls=true"
aws:
enabled: true
region: <REGION>
region: <region>
endpoint: <aurora-endpoint>
roles:
- role: arn:aws:iam::<DB_ACCT_ID>:role/<DB_CONNECT_ROLE_NAME>
- role: arn:aws:iam::<db-account-id>:role/<db-connect-role-name>
----

The `aws.roles` field accepts an array of role ARNs, which are assumed in sequence. This supports chaining multiple role assumptions for more complex cross-account setups.

== Troubleshoot
[[troubleshooting]]
== Troubleshooting

=== is not authorized to perform: sts:AssumeRole
These errors appear in Redpanda Connect pipeline logs. Monitor logs using `+rpk topic consume __redpanda.connect.logs+`.

[[err-sts-assumerole]]
=== `is not authorized to perform: sts:AssumeRole`

The inline assume-role policy is missing from the Redpanda Connect pipeline role, or the tag condition is not met.

. Verify that the database connect role has the `redpanda_scope_redpanda_connect: true` tag.
. Attach the inline policy as described in <<Step 4: Grant the pipeline role cross-account access (Redpanda account)>>.
. Attach the inline policy as described in <<grant-pipeline-cross-account-access,Grant the pipeline role cross-account access>>.

The error clears automatically once the policy is in place. Monitor pipeline logs to confirm:
The error clears automatically after the policy is in place. Monitor pipeline logs to confirm:

[source,bash]
----
rpk topic consume __redpanda.connect.logs --offset end | grep '<PIPELINE_ID>'
rpk topic consume __redpanda.connect.logs --offset end | grep '<pipeline-id>'
----

Look for a message like `postgres_cdc input go active` or `mysql_cdc input go active` to confirm successful authentication.

[[next-steps]]
== Next steps

* xref:components:inputs/postgres_cdc.adoc[`postgres_cdc` input]
* xref:components:inputs/mysql_cdc.adoc[`mysql_cdc` input]

// end::single-source[]