Skip to content

Commit 3df97b1

Browse files
committed
Add Cloud version of recommended IAM policy for Glue integration
1 parent d8976c2 commit 3df97b1

2 files changed

Lines changed: 58 additions & 5 deletions

File tree

modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc

Lines changed: 57 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ If you want to use partitioning, you must specify a custom partition specificati
4646

4747
=== Manual deletion of Iceberg tables
4848

49-
The AWS Glue catalog integration requires Redpanda Iceberg tables to be manually deleted. To manually delete Iceberg tables, you must first set the cluster property config_ref:iceberg_delete,true,properties/cluster-properties[`iceberg_delete`] to `false` when you configure the catalog integration.
49+
The AWS Glue catalog integration does not support automatic deletion of Iceberg tables from Redpanda. To manually delete Iceberg tables in AWS Glue, you must first set the cluster property config_ref:iceberg_delete,true,properties/cluster-properties[`iceberg_delete`] to `false` when you configure the catalog integration.
5050

5151
When `iceberg_delete` is set to `false`, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If your intent is to recreate the topic after deleting it, you are required to delete the table data entirely before recreating the topic.
5252

@@ -56,11 +56,36 @@ ifndef::env-cloud[]
5656
You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog.
5757

5858
For example, you could create a separate IAM policy that manages access to AWS Glue, and attach it to the IAM role that Redpanda also uses to access S3. It is recommended to add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources:
59+
60+
- Root catalog (`catalog`)
61+
- All databases (`database/*`)
62+
- All tables (`table/\*/*`)
63+
64+
Your IAM policy should include a statement similar to the following:
65+
66+
[,json]
67+
----
68+
{
69+
"Version": "2012-10-17",
70+
"Statement": [
71+
{
72+
"Effect": "Allow",
73+
"Action": [
74+
"glue:*"
75+
],
76+
"Resource": [
77+
"arn:aws:glue:<aws-region>:<aws-account-id>:catalog",
78+
"arn:aws:glue:<aws-region>:<aws-account-id>:database/*",
79+
"arn:aws:glue:<aws-region>:<aws-account-id>:table/*/*"
80+
]
81+
}
82+
]
83+
}
84+
----
5985
endif::[]
6086

6187
ifdef::env-cloud[]
6288
You must allow Redpanda access to AWS Glue services in your AWS account. It is recommended to create a new IAM policy or role that manages access to AWS Glue, allowing all AWS Glue API actions (`"glue:*"`) on the following resources:
63-
endif::[]
6489

6590
- Root catalog (`catalog`)
6691
- All databases (`database/*`)
@@ -83,10 +108,38 @@ Your IAM policy should include a statement similar to the following:
83108
"arn:aws:glue:<aws-region>:<aws-account-id>:database/*",
84109
"arn:aws:glue:<aws-region>:<aws-account-id>:table/*/*"
85110
]
111+
},
112+
{
113+
"Effect": "Allow",
114+
"Action": [
115+
"s3:PutObject",
116+
"s3:PutObjectAcl",
117+
"s3:DeleteObject"
118+
],
119+
"Resource": [
120+
"arn:aws:s3:::redpanda-cloud-storage-<redpanda-cluster-id>/redpanda-iceberg-catalog/*"
121+
]
122+
},
123+
{
124+
"Effect": "Allow",
125+
"Action": [
126+
"s3:ListBucket"
127+
],
128+
"Resource": [
129+
"arn:aws:s3:::redpanda-cloud-storage-<redpanda-cluster-id>"
130+
],
131+
"Condition": {
132+
"StringLike": {
133+
"s3:prefix": [
134+
"redpanda-iceberg-catalog/*"
135+
]
136+
}
137+
}
86138
}
87139
]
88140
}
89141
----
142+
endif::[]
90143

91144
For more information on configuring IAM permissions, see the https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html[AWS Glue documentation^].
92145

@@ -125,13 +178,13 @@ Run `rpk cluster config edit` to update these properties:
125178
----
126179
iceberg_enabled: true
127180
# Glue requires Redpanda Iceberg tables to be manually deleted
128-
iceberg_delete: false
181+
iceberg_delete: false
129182
iceberg_catalog_type: rest
130183
iceberg_rest_catalog_endpoint: https://glue.<glue-region>.amazonaws.com/iceberg
131184
iceberg_rest_catalog_authentication_mode: aws_sigv4
132185
iceberg_rest_catalog_base_location: s3://<bucket-name>/<warehouse-path>
133186
# Use the iceberg_rest_catalog_aws_* properties if you want to
134-
# use separate AWS credentials for the catalog, or delete to reuse S3
187+
# use separate AWS credentials for the catalog, or omit these lines to reuse S3
135188
# (cloud_storage_*) credentials.
136189
# For access using access keys only, use iceberg_rest_catalog_aws_access_key
137190
# and iceberg_rest_catalog_aws_secret_key. For access with an IAM role, use

modules/reference/pages/properties/cluster-properties.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2265,7 +2265,7 @@ Source of AWS credentials for Iceberg REST catalog SigV4 authentication. If not
22652265
endif::[]
22662266

22672267
ifdef::env-cloud[]
2268-
Source of AWS credentials for Iceberg REST catalog SigV4 authentication. If using `iceberg_rest_catalog_aws_access_key` and `iceberg_rest_catalog_aws_secret_key` for Glue catalog authentication, you must set this property to `config_file`.
2268+
Source of AWS credentials for Iceberg REST catalog SigV4 authentication. If providing explicit credentials using `iceberg_rest_catalog_aws_access_key` and `iceberg_rest_catalog_aws_secret_key` for Glue catalog authentication, you must set this property to `config_file`.
22692269
endif::[]
22702270

22712271
*Accepted values*: `aws_instance_metadata`, `azure_aks_oidc_federation`, `azure_vm_instance_metadata`, `config_file`, `gcp_instance_metadata`, `sts`.

0 commit comments

Comments
 (0)