Skip to content

Latest commit

 

History

History
20 lines (15 loc) · 12.5 KB

File metadata and controls

20 lines (15 loc) · 12.5 KB

DestinationS3DataLake

Defines the configurations required to connect to an Iceberg catalog, including warehouse location, main branch name, and catalog type specifics.

⚠️ DEPRECATED: Typed connector configuration models are deprecated and will be removed in v1.1.0. Pass configuration as a plain dict with a required 'destinationType' key instead..

Fields

Field Type Required Description Example
access_key_id Optional[str] The AWS Access Key ID with permissions for S3 and Glue operations.
catalog_type models.CatalogType ✔️ Specifies the type of Iceberg catalog (e.g., NESSIE, GLUE, REST, POLARIS) and its associated configuration.
destination_type models.S3DataLake ✔️ N/A
main_branch_name Optional[str] The primary or default branch name in the catalog. Most query engines will use "main" by default. See Iceberg documentation for more information.
s3_bucket_name str ✔️ The name of the S3 bucket that will host the Iceberg data.
s3_bucket_region models.DestinationS3DataLakeS3BucketRegion ✔️ The region of the S3 bucket. See here for all region codes. us-east-1
s3_endpoint Optional[str] Your S3 endpoint url. Read more here
secret_access_key Optional[str] The AWS Secret Access Key paired with the Access Key ID for AWS authentication.
warehouse_location str ✔️ The root location of the data warehouse used by the Iceberg catalog. Typically includes a bucket name and path within that bucket. For AWS Glue and Nessie, must include the storage protocol (such as "s3://" for Amazon S3). s3://your-bucket/path/to/store/files/in