|
| 1 | +--- |
| 2 | +title: "Glue" |
| 3 | +--- |
| 4 | + |
| 5 | +The AWS Glue integration in Elementary will allow you to automatically monitor volume & freshness anomalies in your Iceberg tables in Glue, by continouously syncing metadata about Iceberg snapshots. |
| 6 | + |
| 7 | +<Note> |
| 8 | + If you connected Amazon Athena as your DWH integration - there's no need to add a Glue integration, as the Athena integration already |
| 9 | + contains all the required access for Glue. |
| 10 | +</Note> |
| 11 | + |
| 12 | +## AWS Setup |
| 13 | + |
| 14 | +### 1. Create Required IAM Policy |
| 15 | + |
| 16 | +First, you'll need to create an IAM policy with the following permissions: |
| 17 | +- **GluePermissions**: Enables reading metadata about tables in your Glue catalog. |
| 18 | +- **S3IcebergMetadataReadAccess**: Grants access metadata-only access to files of your Iceberg tables. These metadata files contain statistics about Iceberg snapshots, |
| 19 | +such as update cadence and row count changes. |
| 20 | + |
| 21 | +Here is an example of a JSON policy: |
| 22 | +```json |
| 23 | +{ |
| 24 | + "Version": "2012-10-17", |
| 25 | + "Statement": [ |
| 26 | + { |
| 27 | + "Sid": "GluePermissions", |
| 28 | + "Effect": "Allow", |
| 29 | + "Action": [ |
| 30 | + "glue:GetTable", |
| 31 | + "glue:GetTables" |
| 32 | + ], |
| 33 | + "Resource": "*" |
| 34 | + }, |
| 35 | + { |
| 36 | + "Sid": "S3IcebergMetadataReadAccess", |
| 37 | + "Effect": "Allow", |
| 38 | + "Action": [ |
| 39 | + "s3:GetObject" |
| 40 | + ], |
| 41 | + "Resource": [ |
| 42 | + "arn:aws:s3:::your-iceberg-tables-bucket/*metadata.json" |
| 43 | + ] |
| 44 | + } |
| 45 | + ] |
| 46 | +} |
| 47 | +``` |
| 48 | + |
| 49 | +### 2. Choose Authentication Method |
| 50 | + |
| 51 | +Elementary supports two authentication methods for connecting to Glue: |
| 52 | + |
| 53 | +#### Option 1: AWS Role Authentication (Recommended) |
| 54 | + |
| 55 | +This is the recommended approach as it provides better security and follows AWS best practices. [Learn more about AWS IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). |
| 56 | + |
| 57 | +1. **Create an IAM Role**: |
| 58 | + - Go to AWS IAM Console |
| 59 | + - Create a new role |
| 60 | + - Select "Another AWS account" as the trusted entity |
| 61 | + - Enter Elementary's AWS account ID: `743289191656` |
| 62 | + - (Optional but recommended) Enable "Require external ID" and set a value |
| 63 | + - Attach the policy created in step 1 |
| 64 | + |
| 65 | +2. **Note down the following information**: |
| 66 | + - Role ARN |
| 67 | + - External ID (if you enabled it) [Learn more about external IDs](https://aws.amazon.com/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/). |
| 68 | + |
| 69 | +#### Option 2: Access Key Authentication |
| 70 | + |
| 71 | +This method is less secure as it requires permanent credentials. We recommend using AWS Role authentication instead. |
| 72 | + |
| 73 | +1. **Create an IAM User**: |
| 74 | + - Go to AWS IAM Console |
| 75 | + - Create a new user, that will be used by elementary to connect to Glue |
| 76 | + - Enable programmatic access |
| 77 | + - Attach the policy created in step 1 |
| 78 | + |
| 79 | +2. **Note down the following information**: |
| 80 | + - AWS Access Key ID of the new elementary glue user |
| 81 | + - AWS Secret Access Key of the new elementary glue user |
| 82 | + |
| 83 | +## Elementary Configuration |
| 84 | + |
| 85 | +Navigate to the **Account settings > Environments** and choose the environment to which you would like to connect AWS Glue. |
| 86 | +Under the "Metadata Layer" section, please choose Glue. |
| 87 | + |
| 88 | +### Connection Settings |
| 89 | + |
| 90 | +Regardless of the authentication method you choose, you'll need to provide: |
| 91 | + |
| 92 | +- **Connection Name**: A descriptive name for your connection (e.g. "Datalake"). Needs to be unique if you're adding more than one metadata integration. |
| 93 | +- **Region**: The AWS region where your Glue catalog is located |
| 94 | + |
| 95 | +### Authentication Details |
| 96 | + |
| 97 | +Based on your chosen authentication method: |
| 98 | + |
| 99 | +#### If using AWS Role Authentication: |
| 100 | +- Select "AWS Role" as the authentication method |
| 101 | +- Enter your role ARN |
| 102 | +- Enter your external ID (if you enabled it) |
| 103 | + |
| 104 | +#### If using Access Key Authentication: |
| 105 | +- Select "Access Key" as the authentication method |
| 106 | +- Enter your AWS Access Key ID |
| 107 | +- Enter your AWS Secret Access Key |
0 commit comments