Missing AWS Profile Support in PyIceberg

[thomas-pfeiffer]

Feature Request / Improvement

Feature: Missing AWS Profile Support in PyIceberg / PyIceberg should support AWS profiles

Description:
When working with multiple AWS configs / credentials in parallel, AWS profiles are a convenient way to achieve this. Ideally, PyIceberg should therefore also support AWS profiles, which it currently does not.

Current state (as of writing - pyIceberg v0.10.0):

• The Glue part of the GlueCatalog can be configured to use the profile by specifying the Glue client explicitly in the Glue Catalog or via glue.profile-name config parameter:

from boto3 import Session
...
catalog = GlueCatalog(name="your_glue_catalog",client=Session(profile_name="your_aws_profile").client("glue"),...)

or

catalog = GlueCatalog(
    name="your_glue_catalog",
    **{ 
        "glue.profile-name": "your_aws_profile",
        ...
    },
)

• For fsspec backends, AWS profile support is generally available (see https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem and https://github.com/fsspec/s3fs/blob/aceea3a4985f667979e4d8a5a5b8eeddaf23b7be/s3fs/core.py#L229), but it's not implemented in PyIceberg (see

  iceberg-python/pyiceberg/io/fsspec.py

  Line 199 in e07296e

  fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)

  ). To change that we would need to set the session parameter of the S3FileSystem explicitly:

from s3fs import S3FileSystem
from aiobotocore.session import AioSession
...
fs = S3FileSystem(session=AioSession(profile="your_aws_profile"),...)

• For PyArrow backend, the AWS profile support is not yet available, but they do have an enhancement ticket for it (see [Python][C++] Add Profile support to S3FileSystem arrow#47880). Once AWS profile is supported in PyArrow it can be implemented in PyIceberg as well, I assume.

Workaround for this feature gap:

session = Session(profile_name="your_aws_profile")
credentials = session.get_credentials()  
if credentials is None:
    raise ValueError("Could not retrieve credentials for profile")
catalog = GlueCatalog(
    name="your_glue_catalog",
    **{ 
        "client.access-key-id": credentials.access_key,
        "client.secret-access-key": credentials.secret_key,
        "client.session-token": credentials.token,
        ...
    },
)

To-Be / Expected Behavior:

1. PyIceberg should have a new client.profile-name and s3.profile-name configuration parameter (next to existing glue.profile-name.
2. New client.profile-name should also set glue.profile-name (same behaviour as for all the other unified AWS credentials).
3. For now, AWS profile support should be implemented for fsspec backend and client.profile-name and s3.profile-name should only be supported when using fsspec backend ("py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO").
4. Once PyArrow supports AWS profile names (see [Python][C++] Add Profile support to S3FileSystem arrow#47880), AWS profile support should be implemented for PyArrow backend as well and client.profile-name and s3.profile-name should be fully supported.

Remark: I found this feature gap with the GlueCatalog; it might be that the RestCatalog is equally affected, but not sure.
Issues possibly related to this issue: #570, #1207, #2657

[stats-dev]

Is this issue working on? I want to work on this issue.

[thomas-pfeiffer]

First of all, thanks to @stats-dev for the fix and apologies for the late reply. Few follow-up questions:

As per my understanding of #2948, the new s3.profile-name and client.profile-name are only supported in FsspecFileIO, but not in PyArrowFileIO. @stats-dev and @kevinjqliu - Is that your understanding as well?

(My reasoning being that #2948 didn't change pyiceberg/io/pyarrow.py and the fact that apache/arrow#47880 is still open.)

• If so, do we need to add a remark in the documentation, that s3.profile-name and client.profile-name are only supported when using FsspecFileIO at least for now?
• @kevinjqliu Should I create a new, follow-up issue for the pyArrow part? Or what's the preferred way to track this? (The issue would be effectively blocked till pyArrow implements the AWS profile feature on their side.)

For PyArrow backend, the AWS profile support is not yet available, but they do have an enhancement ticket for it (see apache/arrow#47880). Once AWS profile is supported in PyArrow it can be implemented in PyIceberg as well, I assume.

[stats-dev]

Thanks for the thoughtful follow-up.

Yes — that matches my understanding as well. In #2948 we only added support for s3.profile-name and client.profile-name on the fsspec backend (FsspecFileIO), and PyArrowFileIO is unchanged. So AWS profile selection is not supported on the PyArrow backend yet, and it’s currently blocked on issue #47880.

I agree we should add a short note to the docs clarifying that these options are currently supported only when using FsspecFileIO, maybe making another issue/PR?
Creating a follow-up issue to track PyArrowFileIO support (blocked by apache/arrow #47880) also sounds like the right way to track this.

[thomas-pfeiffer]

@stats-dev sounds good.

FYI - I created #3065 as the follow up ticket to expand the feature to the PyArrrow backend and #3066 for the documentation fix.