feat(rest): support rest catalog with database/table operations and snapshot listing

[SteNicholas]

Purpose

Linked issue: #92

Add a REST catalog implementation, selected via the metastore=rest option, that talks the Paimon REST catalog open API:

• HttpClient: blocking libcurl client with exponential-backoff retries — 429/503 responses are retried for all methods, transport errors only for idempotent methods, and a numeric Retry-After response header takes precedence over the backoff.
• RestApi: HTTP + JSON layer with /v1/config option merging (overrides > client options > defaults), header. options sent as request headers, paged listing, and error mapping to Status (404 → NotExist, 409 → Exist, 400 → Invalid, 501 → NotImplemented) including the x-request-id in error messages.
• RestCatalog: database/table create/drop/rename/list/get, snapshot listing, table-default. option defaults on table creation, system/branch table checks, and conversion of the server table response into a TableSchema with the computed highestFieldId.
• Bear token authentication provider (token.provider=bear).

libcurl is used from the system rather than bundled, so the new CMake option PAIMON_ENABLE_REST defaults to OFF, consistent with the other optional components that need something outside the bundled third-party toolchain (jindo, lance, lumina, lucene, tantivy). CI enables it explicitly in ci/scripts/build_paimon.sh and installs the libcurl development package, so the REST code and its tests are still built and run on every job. With the option off, Catalog::Create reports that metastore=rest requires a build with PAIMON_ENABLE_REST=ON.

Follow-ups (not in this PR): DLF signing, data-token file IO, branch snapshot listing, and the remaining endpoints (alter/commit/views/partitions/tags/functions).

Tests

New paimon-rest-test binary (40 cases, registered under the unittest label so it runs in CI), all against an in-process mock HTTP/REST server:

• HttpClientTest: uri normalization, query encoding, retry classification (429/503 retried, 404 not, transport errors only for idempotent methods), Retry-After precedence, retry exhaustion.
• RestUtilTest / ResourcePathsTest / RestMessagesTest: url encoding and decoding (including the lenient handling of malformed escapes), prefix extraction, resource path building, JSON round trips of all request/response messages, config merge with null-value filtering.
• RestCatalogTest: client-side option validation (missing uri/token, unsupported token provider) and end-to-end catalog operations including pagination, error paths (ignore_if_exists / ignore_if_not_exists), client/server headers, system tables, branch tables, nested-schema highestFieldId, and snapshot listing with sorting.
• RestApiErrorTest: http-status-to-Status mapping including the x-request-id.

Also verified locally that the build still succeeds with PAIMON_ENABLE_REST=OFF (the default, which CI does not exercise), that clang-tidy is clean on the new sources, and ran pre-commit on all changed files.

API and Format

• New public options in include/paimon/defs.h: Options::METASTORE, Options::URI, Options::TOKEN, Options::TOKEN_PROVIDER (all PAIMON_EXPORTed via the existing Options struct).
• Catalog::Create now dispatches on the metastore option (filesystem remains the default; unknown values are rejected).
• No storage format changes. The wire protocol is the Paimon REST catalog open API.

Documentation

docs/source/building.rst lists -DPAIMON_ENABLE_REST=ON under the optional components, including its libcurl requirement. The catalog itself is configured through the existing options mechanism (metastore=rest, uri, token.provider, token). No standalone documentation page is added in this PR.

Generative AI tooling

Generated-by: Claude Code (claude-fable-5)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds a new REST-backed catalog implementation selectable via metastore=rest, including HTTP client + auth + REST API layers plus unit tests and build/CI wiring.

Changes:

• Introduces HttpClient, RestApi, RestCatalog, auth provider, message models, and URL/path utilities for the REST catalog protocol.
• Adds an in-process mock REST server and a new REST unit test binary with coverage across retry logic, message round-trips, and catalog operations.
• Extends catalog factory/options and updates CMake/CI to optionally build/link REST support (libcurl) and run REST tests.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/paimon/rest/rest_util.h	Declares REST helper utilities (URL encode/decode, JSON helpers, prefix extraction).
src/paimon/rest/rest_util.cpp	Implements URL encoding/decoding, prefix extraction, and rapidjson serialization helpers.
src/paimon/rest/rest_messages_test.cpp	Adds unit tests for REST utilities, paths, and JSON message round-trips.
src/paimon/rest/rest_messages.h	Adds REST protocol request/response models (JSON-serializable).
src/paimon/rest/rest_messages.cpp	Implements JSON serialization/deserialization and config merge behavior.
src/paimon/rest/rest_catalog_test.cpp	End-to-end tests for REST catalog behavior against a mock server.
src/paimon/rest/rest_catalog.h	Declares RestCatalog implementing database/table operations + snapshot listing.
src/paimon/rest/rest_catalog.cpp	Implements REST catalog behavior and schema conversion/highestFieldId logic.
src/paimon/rest/rest_auth.h	Declares REST auth provider interface and bearer-token provider.
src/paimon/rest/rest_auth.cpp	Implements bearer-token header injection and provider selection via options.
src/paimon/rest/rest_api.h	Declares REST API layer (HTTP+JSON) and error mapping to Status.
src/paimon/rest/rest_api.cpp	Implements config fetching/merging, request execution, pagination, and error mapping.
src/paimon/rest/resource_paths.h	Declares URL path builder for REST endpoints with optional prefix.
src/paimon/rest/resource_paths.cpp	Implements REST endpoint path construction with URL-encoded segments.
src/paimon/rest/mock_rest_server.h	Declares a minimal blocking HTTP server for REST unit tests.
src/paimon/rest/mock_rest_server.cpp	Implements the mock HTTP server (parsing, dispatch, responses).
src/paimon/rest/http_client_test.cpp	Adds unit tests for URI normalization, query encoding, and retry behavior.
src/paimon/rest/http_client.h	Declares blocking libcurl client with retry/backoff behavior.
src/paimon/rest/http_client.cpp	Implements libcurl execution, retry classification, backoff, and header parsing.
src/paimon/core/snapshot.h	Adds SnapshotInfo include and Snapshot::ToSnapshotInfo() declaration.
src/paimon/core/snapshot.cpp	Implements Snapshot::ToSnapshotInfo() commit-kind mapping and field transfer.
src/paimon/core/catalog/file_system_catalog.cpp	Refactors snapshot listing to reuse Snapshot::ToSnapshotInfo().
src/paimon/core/catalog/catalog.cpp	Adds metastore dispatch for filesystem vs rest (guarded by PAIMON_ENABLE_REST).
src/paimon/common/defs.cpp	Defines new option keys: metastore, uri, token, token.provider.
src/paimon/CMakeLists.txt	Adds REST sources/linking behind PAIMON_ENABLE_REST and registers REST tests.
include/paimon/defs.h	Exposes new public options for selecting/configuring the REST catalog.
ci/scripts/build_paimon.sh	Forces -DPAIMON_ENABLE_REST=ON in CI build script.
CMakeLists.txt	Adds PAIMON_ENABLE_REST option, finds libcurl, and defines compile macro.
.github/workflows/gcc8_test.yaml	Installs libcurl dev package for GCC8 CI.
.github/workflows/build_and_test.yaml	Installs libcurl dev package for general CI builds/tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.