Skip to content

S3, GCS, Azure Blob Destinations #30

Merged
kmacrow merged 2 commits into
masterfrom
object-stores
Sep 2, 2025
Merged

S3, GCS, Azure Blob Destinations #30
kmacrow merged 2 commits into
masterfrom
object-stores

Conversation

@kmacrow
Copy link
Copy Markdown
Collaborator

@kmacrow kmacrow commented Aug 15, 2025

  • add append-only hive table format destinations for s3, gcs, azure blob
  • introduces a format connection param for the object store connectors that configures the format to write:
    • staging: an internal format optimized for staging data for load/unload between other data stores
    • hive: Hive compatible table format with a dt=<transfer_date> partition
    • future I imagine we can add iceberg, delta, etc. as supported formats
  • integration test scaffolding to demo how they work, but these need to be extended
  • new docs pages for connectors

@kmacrow kmacrow merged commit 5101a5d into master Sep 2, 2025
3 checks passed
@kmacrow kmacrow deleted the object-stores branch September 2, 2025 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants