feature(s3tables): read-only Amazon S3 Tables support via icebird/s3tables subpath#29
Merged
platypii merged 9 commits intoJul 11, 2026
Conversation
The-Alchemist
force-pushed
the
cursor/add-s3-tables-support
branch
from
43d5f83 to
e3bdbc3
Compare
The-Alchemist
changed the title
Adds an optional `icebird/s3tables` subpath that connects to the S3 Tables Iceberg REST catalog and reads table data. The core `icebird` package stays AWS-free: requests are signed with icebird's own SigV4 implementation, and the only optional peer dependency is `@aws-sdk/credential-providers` (imported lazily, and only when resolving the default credential chain — explicit credentials need no AWS SDK at all). - New `icebird/s3tables` subpath export and `src/aws/*` modules. - Generic `signRequest` hook on the REST catalog (`restCatalogConnect` / `restFetch`) so signing stays cloud-agnostic. - Extract the hand-rolled SigV4 primitives into `src/sigv4.js` and reuse them for both S3 data reads (service `s3`) and the S3 Tables catalog (`s3tables`). Fixes found while reading a live S3 Tables bucket: - double-URI-encode the canonical path for non-`s3` services (S3 Tables 403'd on catalog paths containing the URL-encoded warehouse prefix) - use the regional S3 endpoint so non us-east-1 buckets don't 301 redirect - decode boxed primitives, maps, and enums in the Avro reader, which S3 Tables manifests use Signed-off-by: Karl Pietrzak <karl@medplum.com>
The-Alchemist
force-pushed
the
cursor/add-s3-tables-support
branch
from
e3bdbc3 to
4b9e629
Compare
Signed-off-by: Karl Pietrzak <karl@medplum.com>
The v0.8.12 characterization test expects multiple aggregates on one column to share a single scanColumn pass, but that behavior only landed in squirreling 0.12.26. Signed-off-by: Karl Pietrzak <karl@medplum.com>
The-Alchemist
force-pushed
the
cursor/add-s3-tables-support
branch
from
458e1a6 to
d0b84ca
Compare
platypii
force-pushed
the
cursor/add-s3-tables-support
branch
from
d0b84ca to
805f700
Compare
platypii
force-pushed
the
cursor/add-s3-tables-support
branch
from
805f700 to
b0be936
Compare
platypii
approved these changes
Jul 11, 2026
platypii
left a comment
platypii
left a comment
Contributor
There was a problem hiding this comment.
Looks good thanks @The-Alchemist !
I tested this against real s3tables. First confirmed that master failed. Then your branch, and indeed looks good!
I pushed some modifications to the branch, hope that was ok.
- merge master and fix conflicts
- fix lint error
- remove s3tables smoke script
- move AWS typedefs to src/aws/types.d.ts
- don't require tableBucketArn in s3TablesResolver options
Contributor
|
Published in icebird 0.8.15 thanks again @The-Alchemist! |
Contributor
Author
|
np, @platypii ! the next step for us is a version with S3 Tables support that also understands Lake Formation grants (https://docs.aws.amazon.com/lake-formation/latest/dg/enable-s3-tables-catalog-integration.html), the recommended approach for handling fine-grained permissions. unfortunately, that's another dependency ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
icebird/s3tablessubpath that connects to the Amazon S3 Tables Iceberg REST catalog and reads table data. The coreicebirdpackage stays AWS-free.icebird/s3tablessubpath export backed bysrc/aws/*modules.src/sigv4.js) — no AWS signing SDK.@aws-sdk/credential-providers, used solely to resolve the default AWS credential chain. It is imported lazily, so passing explicit credentials needs no AWS SDK at all.signRequesthook on the REST catalog (restCatalogConnect/restFetch) — a cloud-agnostic per-request auth hook.src/s3.jsintosrc/sigv4.js, reused for both S3 data reads (services3) and the S3 Tables catalog (services3tables).Fixes found while reading a live S3 Tables bucket
s3services. S3 Tables catalog paths carry the URL-encoded warehouse prefix; S3 uses single-encoding, every other service double-encodes.us-east-1(incl. S3 Tables--table-s3buckets) don't redirect.{"type":"string"}), maps, and enums, which S3 Tables manifests emit.Questions
awsas a dependency to the whole icebird packagescripts/s3tables-smoke.mjsin-tree