Skip to content

Commit 9a453dd

Browse files
Fokkosungwy
Fokko
authored and
sungwy
committed
Allow passing in ARN Role and Session name (apache#1296)
1 parent ca077d0 commit 9a453dd

File tree

3 files changed

+25
-9
lines changed

3 files changed

+25
-9
lines changed

‎mkdocs/docs/configuration.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -101,18 +101,20 @@ For the FileIO there are several configuration options available:
101101

102102
<!-- markdown-link-check-disable -->
103103

104-
| Key | Example | Description |
105-
| -------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
104+
| Key | Example | Description |
105+
|----------------------|----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
106106
| s3.endpoint | <https://10.0.19.25/> | Configure an alternative endpoint of the S3 service for the FileIO to access. This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. |
107-
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
108-
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
109-
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
110-
| s3.signer | bearer | Configure the signature version of the FileIO. |
107+
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
108+
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
109+
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
110+
| s3.session-name | session | An optional identifier for the assumed role session. |
111+
| s3.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
112+
| s3.signer | bearer | Configure the signature version of the FileIO. |
111113
| s3.signer.uri | <http://my.signer:8080/s3> | Configure the remote signing uri if it differs from the catalog uri. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. |
112-
| s3.signer.endpoint | v1/main/s3-sign | Configure the remote signing endpoint. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. (default : v1/aws/s3/sign). |
113-
| s3.region | us-west-2 | Sets the region of the bucket |
114+
| s3.signer.endpoint | v1/main/s3-sign | Configure the remote signing endpoint. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. (default : v1/aws/s3/sign). |
115+
| s3.region | us-west-2 | Sets the region of the bucket |
114116
| s3.proxy-uri | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO. |
115-
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
117+
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
116118

117119
<!-- markdown-link-check-enable-->
118120

‎pyiceberg/io/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,8 @@
6060
AWS_ACCESS_KEY_ID = "client.access-key-id"
6161
AWS_SECRET_ACCESS_KEY = "client.secret-access-key"
6262
AWS_SESSION_TOKEN = "client.session-token"
63+
AWS_ROLE_ARN = "aws.role-arn"
64+
AWS_SESSION_NAME = "aws.session-name"
6365
S3_ENDPOINT = "s3.endpoint"
6466
S3_ACCESS_KEY_ID = "s3.access-key-id"
6567
S3_SECRET_ACCESS_KEY = "s3.secret-access-key"
@@ -70,6 +72,8 @@
7072
S3_SIGNER_URI = "s3.signer.uri"
7173
S3_SIGNER_ENDPOINT = "s3.signer.endpoint"
7274
S3_SIGNER_ENDPOINT_DEFAULT = "v1/aws/s3/sign"
75+
S3_ROLE_ARN = "s3.role-arn"
76+
S3_SESSION_NAME = "s3.session-name"
7377
HDFS_HOST = "hdfs.host"
7478
HDFS_PORT = "hdfs.port"
7579
HDFS_USER = "hdfs.user"

‎pyiceberg/io/pyarrow.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,9 @@
8585
from pyiceberg.io import (
8686
AWS_ACCESS_KEY_ID,
8787
AWS_REGION,
88+
AWS_ROLE_ARN,
8889
AWS_SECRET_ACCESS_KEY,
90+
AWS_SESSION_NAME,
8991
AWS_SESSION_TOKEN,
9092
GCS_DEFAULT_LOCATION,
9193
GCS_ENDPOINT,
@@ -101,7 +103,9 @@
101103
S3_ENDPOINT,
102104
S3_PROXY_URI,
103105
S3_REGION,
106+
S3_ROLE_ARN,
104107
S3_SECRET_ACCESS_KEY,
108+
S3_SESSION_NAME,
105109
S3_SESSION_TOKEN,
106110
FileIO,
107111
InputFile,
@@ -362,6 +366,12 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste
362366
if connect_timeout := self.properties.get(S3_CONNECT_TIMEOUT):
363367
client_kwargs["connect_timeout"] = float(connect_timeout)
364368

369+
if role_arn := get_first_property_value(self.properties, S3_ROLE_ARN, AWS_ROLE_ARN):
370+
client_kwargs["role_arn"] = role_arn
371+
372+
if session_name := get_first_property_value(self.properties, S3_SESSION_NAME, AWS_SESSION_NAME):
373+
client_kwargs["session_name"] = session_name
374+
365375
return S3FileSystem(**client_kwargs)
366376
elif scheme in ("hdfs", "viewfs"):
367377
from pyarrow.fs import HadoopFileSystem

0 commit comments

Comments
 (0)