feat(fs): add lightweight object-store and S3 filesystem support by mrdrivingduck · Pull Request #433 · alibaba/paimon-cpp

mrdrivingduck · 2026-07-19T17:20:58Z

Background

I started this work because I wanted duckdb-paimon to query Paimon data lakes stored on S3. That required adding S3 filesystem support to paimon-cpp first.

My first implementation used Aws::S3::S3Client. It worked, but the dependency cost was much larger than I expected. Reading a Paimon table only needs a fairly small part of S3: HEAD, paginated LIST, and ranged GET. Pulling in the complete AWS S3 client and its CRT dependency stack felt disproportionate for that job, and it also made the build integration considerably more complicated.

I then looked at how DuckDB's httpfs and Iceberg extensions divide this work. DuckDB keeps the HTTP data path generic and limits the storage-specific code to things such as credentials, request signing, URL construction, and error handling. That seemed like a better direction for paimon-cpp as well.

Based on that, I rewrote the original S3 implementation around a reusable object-store filesystem layer and a small HTTP client abstraction.

Design

The implementation is still being cleaned up, and some names and file boundaries may change before it is ready to merge. But this is the architectural split I expect to keep. I am sharing it now mainly to discuss whether this direction makes sense.

The implementation is organized as follows:

FileSystem
└── ObjectStoreFileSystem
    ├── S3FileSystem
    └── [OSSFileSystem] (WIP)

Shared components
├── ObjectStoreClient
├── HttpClient
├── ObjectStoreInputStream
└── Read-ahead

ObjectStoreFileSystem, ObjectStoreClient, and ObjectStoreInputStream implement the filesystem semantics and reading behavior shared by object stores. S3FileSystem and S3ObjectStoreClient only handle S3-specific URLs, API requests, credentials, signing, and errors.

Object data is transferred by the libcurl-based HttpClient. The AWS C libraries are used only for the credential chain and SigV4 signing, so the implementation does not depend on the complete AWS S3 SDK. All of these interfaces remain internal to paimon-cpp, and the existing public FileSystem interface is unchanged.

OSS experiment

To validate the abstraction against a second provider, I also built a WIP OSS filesystem using Alibaba Cloud OSS C++ SDK v2. It reuses the common filesystem semantics, input stream, range reads, read-ahead, pagination, and error handling; the OSS-specific code mainly adapts credentials, endpoints, requests, and responses.

The OSS implementation is not part of this PR, but it could later replace or coexist with the current Jindo-based path. Jindo is a closed-source binary dependency and is no longer recommended in a number of environments, which makes it less suitable as the project's long-term OSS foundation than the open-source OSS SDK.

Scope

The S3 filesystem in this PR is read-only. It implements the operations needed to read Paimon metadata, manifests, and data files. Write and mutation operations return an explicit unsupported error.

This PR does not add a general-purpose http:// filesystem, and it does not attempt to cover HDFS. HDFS has different filesystem semantics and should remain a separate implementation if it is added later.

Testing

Unit tests cover the shared object-store behavior, reads and read-ahead, error handling, and S3 configuration and signing. I also tested metadata, manifest, and ORC reads against a real S3 Paimon warehouse and verified the integration through duckdb-paimon.

Copilot

Pull request overview

This PR adds a new internal object-store filesystem layer and a read-only S3 filesystem implementation, enabling paimon-cpp to read Paimon warehouses stored on S3 with a lightweight HTTP data path (libcurl) and minimal AWS C libraries for credential discovery and SigV4 signing.

Changes:

Introduce object_store shared components: ObjectStoreFileSystem (read-only semantics + read-ahead) and a libcurl-based HttpClient.
Add S3FileSystem + factory and an S3ObjectStoreClient that implements HEAD/LIST/Range-GET and request signing via AWS C Auth.
Extend build/CI to support an opt-in PAIMON_ENABLE_S3 option and build the minimal AWS auth dependency stack.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
third_party/versions.txt	Adds version pins/checksums for minimal AWS C auth dependency set (and s2n).
src/paimon/fs/s3/s3_file_system.h	Declares S3 filesystem options and construction helpers.
src/paimon/fs/s3/s3_file_system.cpp	Implements S3 object-store client (signing + HEAD/LIST/GET range) and S3 filesystem wrapper.
src/paimon/fs/s3/s3_file_system_test.cpp	Adds unit tests for S3 URL construction, signing, option validation, range reads, and list parsing.
src/paimon/fs/s3/s3_file_system_factory.h	Declares S3 filesystem factory for scheme registration.
src/paimon/fs/s3/s3_file_system_factory.cpp	Implements option validation and registers the S3 filesystem factory.
src/paimon/fs/s3/CMakeLists.txt	Builds the S3 filesystem library/tests when `PAIMON_ENABLE_S3` is enabled.
src/paimon/fs/object_store/object_store_file_system.h	Defines object-store client interface, read-ahead limiter, and filesystem wrapper.
src/paimon/fs/object_store/object_store_file_system.cpp	Implements read-only object-store filesystem semantics and read-ahead input stream.
src/paimon/fs/object_store/object_store_file_system_test.cpp	Adds unit tests covering object-store semantics, pagination, and read-ahead behavior.
src/paimon/fs/object_store/http_client.h	Defines internal HTTP client interface and Curl implementation.
src/paimon/fs/object_store/http_client.cpp	Implements Curl-based HTTP execution with basic retries and header parsing.
src/paimon/fs/object_store/CMakeLists.txt	Builds the shared object-store components and tests behind `PAIMON_ENABLE_S3`.
CMakeLists.txt	Adds `PAIMON_ENABLE_S3` option and wires S3/object_store subdirectories + CURL discovery.
cmake_modules/ThirdpartyToolchain.cmake	Allows `_REVISION=` entries and triggers AWS auth build when S3 is enabled.
cmake_modules/BuildAwsAuth.cmake	Adds ExternalProject build for minimal AWS C Auth dependency chain (and s2n on Linux).
cmake_modules/arrow.diff	Adjusts Arrow’s thrift build args (adds `-DWITH_OPENSSL=OFF`).
ci/scripts/build_paimon.sh	Enables `PAIMON_ENABLE_S3=ON` in CI build script.
.github/workflows/gcc8_test.yaml	Installs libcurl/OpenSSL dev packages needed for S3/AWS auth build.
.github/workflows/build_and_test.yaml	Installs libcurl/OpenSSL dev packages before building.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

+            setenv(name, value->c_str(), 1);
+        } else {
+            unsetenv(name);
+        }


+    S3ObjectStoreClient(const std::map<std::string, std::string>& options,
+                        std::shared_ptr<HttpClient> http_client)
+        : http_client_(std::move(http_client)), credentials_(MakeCredentialsProvider(options)) {
+        region_ = ResolveRegion(options);


+void ReadAheadMemoryLimiter::Release(int64_t size) {
+    std::scoped_lock lock(mutex_);
+    used_ -= size;
+}


+    std::string key = parsed.path;
+    while (!key.empty() && key.front() == '/') {
+        key.erase(key.begin());
+    }


Add a reusable read-only object store layer with a curl-based HTTP transport. Use the AWS C authentication components for credential resolution and request signing while keeping S3 data access independent of the full AWS SDK. Co-authored-by: GPT-5.6 Terra <codex@users.noreply.github.com>

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch 7 times, most recently from bdbb0e8 to 3137230 Compare July 20, 2026 12:45

mrdrivingduck marked this pull request as ready for review July 20, 2026 17:11

Copilot AI review requested due to automatic review settings July 20, 2026 17:11

Copilot started reviewing on behalf of mrdrivingduck July 20, 2026 17:11 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/object_store/http_client.cpp

Comment thread cmake_modules/BuildAwsAuth.cmake Outdated

lxy-9602 requested a review from lucasfang July 21, 2026 05:31

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from 3137230 to e52f0b9 Compare July 21, 2026 06:21

mrdrivingduck requested a review from Copilot July 21, 2026 07:20

Copilot started reviewing on behalf of mrdrivingduck July 21, 2026 07:24 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/s3/s3_file_system_test.cpp Outdated

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from e52f0b9 to 1d2f7d1 Compare July 21, 2026 07:50

mrdrivingduck requested a review from Copilot July 21, 2026 09:22

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot started reviewing on behalf of mrdrivingduck July 21, 2026 10:08 View session

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from 1d2f7d1 to 27ee4b7 Compare July 21, 2026 13:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(fs): add lightweight object-store and S3 filesystem support#433

feat(fs): add lightweight object-store and S3 filesystem support#433
mrdrivingduck wants to merge 1 commit into
alibaba:mainfrom
mrdrivingduck:codex/feat_object_store_s3

mrdrivingduck commented Jul 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mrdrivingduck commented Jul 19, 2026

Background

Design

OSS experiment

Scope

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants