Merge branch 'main' into codex/fetch-boost-from-oss

Author: zjw1111

README.md

@@ -32,10 +32,17 @@ Paimon C++ is a high-performance C++ implementation of [Apache Paimon](https://p
 - **File systems**: file system abstraction with built-in local and Jindo file system support.
 - **File formats**: file format abstraction with built-in ORC, Parquet, and Avro support.
 - **Runtime utilities**: memory pool and thread pool abstractions with default implementations.
-- **AI-Oriented Features**: supports RowTracking and DataEvolution mode and provides Global Index capabilities including bitmap index, B-tree index, DiskANN-based vector search with Lumina, and Lucene-based full-text search.
+- **AI-Oriented Features**: supports RowTracking and DataEvolution mode and provides Global Index
+  capabilities including B-tree index, DiskANN-based vector search with Lumina, and Lucene-based
+  full-text search.
 - **Compatibility**: compatibility with Apache Paimon Java format and communication protocols,
   including commit messages, data splits, and manifests.
 
+> **Bitmap global index compatibility:** Java Paimon now uses a dedicated bitmap global index
+> format instead of the previously shared wrapped bitmap file index format.
+> Paimon C++ therefore currently treats the `bitmap` global index type as unsupported. The legacy
+> implementation remains in the codebase pending migration to the Java-compatible format.
+
 Note: Linux x86_64 and macOS arm64 builds are currently verified.
 
 ## Write And Commit Example

benchmark/CMakeLists.txt

@@ -57,7 +57,6 @@ if(PAIMON_BUILD_BENCHMARKS)
                          ${PAIMON_BENCHMARK_STATIC_LINK_LIBS}
                          test_utils_static
                          Threads::Threads
-                         ${CMAKE_DL_LIBS}
                          ${PAIMON_BENCHMARK_PLATFORM_LINK_LIBS}
                          ${PAIMON_BENCHMARK_LINK_TOOLCHAIN}
                          EXTRA_INCLUDES

cmake_modules/ThirdpartyToolchain.cmake

@@ -1653,6 +1653,9 @@ macro(build_arrow)
 
     # libarrow.a calls dlsym; keep ${CMAKE_DL_LIBS} in the interface so -ldl is placed
     # after libarrow.a on linkers that resolve symbols strictly left-to-right.
+    # Every library that uses dl itself (arrow here; also lucene and jindosdk::nextarch)
+    # declares it on its own interface the same way; consumers inherit it transitively
+    # and must not list ${CMAKE_DL_LIBS} again themselves.
     target_link_libraries(arrow
                           INTERFACE zstd
                                     snappy

docs/source/user_guide/data_types.rst

@@ -217,3 +217,44 @@ and `Arrow DataTypes <https://arrow.apache.org/docs/format/Columnar.html#data-ty
 
        The type can be declared using ``ROW<n0 t0 'd0', n1 t1 'd1', ...>`` where n
        is the unique name of a field, t is the logical type of a field, d is the description of a field.
+
+   * - ``VARIANT``
+     - Struct
+     - Data type of semi-structured data (e.g. JSON). A variant value contains one of:
+       a primitive (e.g. integer, string), an array of variant values, or an object
+       mapping string keys to variant values.
+
+       Variant values are encoded with two binaries following the parquet-format
+       `Variant Binary Encoding <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`_
+       specification (compatible with the Java Paimon / Spark implementation). In
+       C++ Paimon, a variant field is represented in an Arrow schema as
+       ``Struct{value: Binary NOT NULL, metadata: Binary NOT NULL}`` marked with
+       Paimon-specific field metadata; use ``paimon::Variant::ArrowField`` to
+       construct such a field, and ``paimon::Variant`` (``FromJson``/``ToJson``/
+       ``VariantGet``) to build and inspect values.
+
+       Only the parquet file format supports VARIANT columns. VARIANT cannot be
+       used as a primary key, partition key or bucket key, and no predicate
+       pushdown applies to it.
+
+       When writing, variant columns can optionally be *shredded* into typed
+       parquet columns per the parquet-format
+       `Variant Shredding <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`_
+       specification by setting ``variant.shreddingSchema`` to a ROW type JSON
+       whose fields map top-level variant column names to their shredding
+       types. Alternatively, setting ``variant.inferShreddingSchema`` to
+       ``true`` infers a shredding schema per file from the first written rows
+       (tuned by ``variant.shredding.maxSchemaWidth``, which bounds the total
+       number of shredded fields across all variant columns of the schema,
+       ``variant.shredding.maxSchemaDepth``,
+       ``variant.shredding.minFieldCardinalityRatio`` and
+       ``variant.shredding.maxInferBufferRow``). Inference also covers variant
+       columns nested inside ROW columns (variants inside arrays or maps stay
+       unshredded, as in Java Paimon). Readers reassemble shredded columns
+       transparently.
+
+       When reading, instead of the full variant, specific paths can be
+       extracted by replacing the variant column in the read schema with a
+       projection built by ``paimon::VariantAccessBuilder`` (e.g. ``$.a.b`` as
+       BIGINT). For shredded files only the required typed sub-columns are
+       read.

include/paimon/data/variant.h

@@ -0,0 +1,197 @@
+/*
+ * Copyright 2026-present Alibaba Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <optional>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+
+#include "paimon/memory/memory_pool.h"
+#include "paimon/predicate/literal.h"
+#include "paimon/result.h"
+#include "paimon/visibility.h"
+
+struct ArrowArray;
+struct ArrowSchema;
+
+namespace paimon {
+
+/// Arguments controlling how a variant value is cast to a target type in `Variant::VariantGet`.
+struct PAIMON_EXPORT VariantCastArgs {
+    /// Whether an invalid cast fails the call (true) or yields SQL NULL (false).
+    bool fail_on_error = true;
+    /// The time zone used when rendering TIMESTAMP values. Supported forms are `UTC`/`Z`/`GMT`,
+    /// fixed offsets such as `+08:00`, and IANA region ids such as `Asia/Shanghai`.
+    std::string zone_id = "UTC";
+};
+
+/// A Variant represents a type that contains one of: 1) Primitive: A type and corresponding
+/// value (e.g. INT, STRING); 2) Array: An ordered list of Variant values; 3) Object: An
+/// unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not
+/// contain duplicate keys.
+///
+/// A Variant is encoded with 2 binaries: the value and the metadata, following the parquet
+/// Variant Binary Encoding specification (compatible with the Java / Spark implementation). The
+/// encoding allows representation of semi-structured data (e.g. JSON) in a form that can be
+/// efficiently queried by path.
+///
+/// In an Arrow schema, a Variant field is represented as
+/// `struct<value: binary not null, metadata: binary not null>` marked with Paimon-specific field
+/// metadata; use `Variant::ArrowField` to construct such a field.
+class PAIMON_EXPORT Variant {
+ public:
+    ~Variant();
+
+    /// Parses a JSON string as a Variant (duplicate object keys are rejected).
+    ///
+    /// @param json The JSON document text.
+    /// @param pool The memory pool used for the variant buffers.
+    /// @return A result containing the created variant or an error.
+    static Result<std::unique_ptr<Variant>> FromJson(const std::string& json,
+                                                     const std::shared_ptr<MemoryPool>& pool);
+
+    /// Creates a Variant from already-encoded value and metadata binaries (copied into `pool`).
+    ///
+    /// @param value The variant value binary.
+    /// @param value_length The length of the value binary.
+    /// @param metadata The variant metadata binary.
+    /// @param metadata_length The length of the metadata binary.
+    /// @param pool The memory pool used for the variant buffers.
+    /// @return A result containing the created variant or an error.
+    static Result<std::unique_ptr<Variant>> Create(const char* value, uint64_t value_length,
+                                                   const char* metadata, uint64_t metadata_length,
+                                                   const std::shared_ptr<MemoryPool>& pool);
+
+    /// The variant value binary. The view remains valid as long as this variant exists.
+    std::string_view Value() const;
+
+    /// The variant metadata binary. The view remains valid as long as this variant exists.
+    std::string_view Metadata() const;
+
+    /// The size of the variant in bytes (value size + metadata size).
+    int64_t SizeInBytes() const;
+
+    /// Stringifies the variant in JSON format.
+    ///
+    /// @param zone_id The time zone used when rendering TIMESTAMP values.
+    /// @return A result containing the JSON text or an error.
+    Result<std::string> ToJson(const std::string& zone_id = "UTC") const;
+
+    /// Extracts a sub-variant value according to a path which starts with a `$`, e.g. `$.key`,
+    /// `$['key']`, `$["key"]`, `$.array[0]`, and casts the value to the target type.
+    ///
+    /// @param path The extraction path.
+    /// @param target_type The target Arrow type (C data interface, consumed by the call); only
+    ///        scalar types are supported. Use `VariantGetArrow` for nested targets.
+    /// @param cast_args Cast behavior arguments.
+    /// @return A result containing the extracted literal, or nullopt for SQL NULL (unmatched
+    ///         path, variant null, or an invalid cast with `fail_on_error == false`).
+    Result<std::optional<Literal>> VariantGet(const std::string& path,
+                                              struct ArrowSchema* target_type,
+                                              const VariantCastArgs& cast_args) const;
+
+    /// Extracts a sub-variant value according to a path which starts with a `$` and casts the
+    /// value to the (possibly nested) target field type.
+    ///
+    /// In addition to scalar types, the target may be a STRUCT (cast from a variant object,
+    /// children matched by field name; unmatched children are null), a MAP with string keys
+    /// (cast from a variant object), a LIST (cast from a variant array), or a variant-marked
+    /// field created by `Variant::ArrowField` (the sub-variant is deeply re-encoded).
+    ///
+    /// @param path The extraction path.
+    /// @param target_field The target Arrow field (C data interface, consumed by the call).
+    /// @param cast_args Cast behavior arguments.
+    /// @return A result containing a length-1 Arrow array of the target type whose single slot
+    ///         holds the result; a null slot represents SQL NULL (unmatched path, variant null,
+    ///         or an invalid cast with `fail_on_error == false`).
+    Result<std::unique_ptr<struct ArrowArray>> VariantGetArrow(
+        const std::string& path, struct ArrowSchema* target_field,
+        const VariantCastArgs& cast_args) const;
+
+    /// Extracts a sub-variant value according to a path which starts with a `$` and renders it
+    /// as JSON text.
+    ///
+    /// @param path The extraction path.
+    /// @param zone_id The time zone used when rendering TIMESTAMP values.
+    /// @return A result containing the JSON text, or nullopt if the path does not match.
+    Result<std::optional<std::string>> VariantGetJson(const std::string& path,
+                                                      const std::string& zone_id = "UTC") const;
+
+    /// Creates an Arrow field definition for the Variant type.
+    ///
+    /// This function constructs an Arrow Field (internally
+    /// `struct<value: binary not null, metadata: binary not null>`) and exports it to the C data
+    /// interface structure `::ArrowSchema`. It automatically injects Paimon-specific metadata to
+    /// identify the field as a VARIANT.
+    ///
+    /// @param field_name The name of the Arrow field.
+    /// @param nullable Whether the field is nullable.
+    /// @param metadata A map of key-value metadata to be attached to the field.
+    /// @return A result containing a unique pointer to the generated `::ArrowSchema` or an error.
+    static Result<std::unique_ptr<struct ArrowSchema>> ArrowField(
+        const std::string& field_name, bool nullable = true,
+        std::unordered_map<std::string, std::string> metadata = {});
+
+ private:
+    class Impl;
+
+    explicit Variant(std::unique_ptr<Impl>&& impl);
+
+    std::unique_ptr<Impl> impl_;
+};
+
+/// Builds a variant-access projection field: a struct field that replaces a VARIANT column in
+/// the read schema so that, instead of the full variant, only the described paths are extracted
+/// at read time (reading only the required shredded sub-columns from shredded files).
+///
+/// Example: read `$.age` as INT64 and `$.city` as STRING from variant column `v`:
+///
+///     VariantAccessBuilder builder;
+///     builder.AddField(age_type, "$.age");
+///     builder.AddField(city_type, "$.city");
+///     auto field = builder.Build("v");  // use in ReadContextBuilder::SetReadSchema
+///
+/// The resulting struct has one child per added field, named by its position ("0", "1", ...).
+class PAIMON_EXPORT VariantAccessBuilder {
+ public:
+    VariantAccessBuilder();
+    ~VariantAccessBuilder();
+
+    /// Adds an extracted field.
+    ///
+    /// @param target_type The Arrow type the extracted value is cast to (C data interface,
+    ///        consumed by the call). Nested targets follow `Variant::VariantGetArrow` semantics.
+    /// @param path The extraction path, e.g. `$.a.b` or `$.array[0]`.
+    /// @param fail_on_error Whether an invalid cast fails the read (true) or yields SQL NULL.
+    /// @param zone_id The time zone used when rendering TIMESTAMP values.
+    Status AddField(struct ArrowSchema* target_type, const std::string& path,
+                    bool fail_on_error = true, const std::string& zone_id = "UTC");
+
+    /// Builds the projection field named `field_name`, to be used in the read schema in place
+    /// of the variant column with the same name.
+    Result<std::unique_ptr<struct ArrowSchema>> Build(const std::string& field_name) const;
+
+ private:
+    class Impl;
+
+    std::unique_ptr<Impl> impl_;
+};
+
+}  // namespace paimon

include/paimon/defs.h

@@ -46,6 +46,7 @@ enum class FieldType {
     MAP = 14,
     STRUCT = 15,
     BLOB = 16,
+    VARIANT = 17,
     UNKNOWN = 128,
 };
 
@@ -414,6 +415,28 @@ struct PAIMON_EXPORT Options {
     /// Only effective when map.storage-layout = shared-shredding.
     static const char MAP_SHARED_SHREDDING_COLUMN_PLACEMENT_POLICY[];
 
+    /// "variant.shreddingSchema" - The Variant shredding schema for writing: a ROW type JSON
+    /// whose fields map variant column names to their shredding types. No default value.
+    static const char VARIANT_SHREDDING_SCHEMA[];
+    /// "parquet.variant.shreddingSchema" - Fallback key of "variant.shreddingSchema".
+    static const char PARQUET_VARIANT_SHREDDING_SCHEMA[];
+    /// "variant.inferShreddingSchema" - Whether to automatically infer the shredding schema when
+    /// writing Variant columns. Default value is "false".
+    static const char VARIANT_INFER_SHREDDING_SCHEMA[];
+    /// "variant.shredding.maxSchemaWidth" - Maximum number of shredded fields allowed in an
+    /// inferred schema. Default value is 300.
+    static const char VARIANT_SHREDDING_MAX_SCHEMA_WIDTH[];
+    /// "variant.shredding.maxSchemaDepth" - Maximum traversal depth in Variant values during
+    /// schema inference. Default value is 50.
+    static const char VARIANT_SHREDDING_MAX_SCHEMA_DEPTH[];
+    /// "variant.shredding.minFieldCardinalityRatio" - Minimum fraction of rows that must contain
+    /// a field for it to be shredded. Fields below this threshold stay in the un-shredded
+    /// Variant binary. Default value is 0.1.
+    static const char VARIANT_SHREDDING_MIN_FIELD_CARDINALITY_RATIO[];
+    /// "variant.shredding.maxInferBufferRow" - Maximum number of rows to buffer for schema
+    /// inference. Default value is 4096.
+    static const char VARIANT_SHREDDING_MAX_INFER_BUFFER_ROW[];
+
     /// "blob-as-descriptor" - Read blob field using blob descriptor rather than blob
     /// bytes. Default value is "false".
     static const char BLOB_AS_DESCRIPTOR[];
@@ -440,13 +463,14 @@ struct PAIMON_EXPORT Options {
     /// path and requires manual configuration by the user. No default value.
     static const char BLOB_VIEW_UPSTREAM_WAREHOUSE[];
     /// "blob-write-null-on-missing-file" - Whether to write NULL for a descriptor BLOB value when
-    /// the referenced file does not exist at write time. When false, the write fails when the
-    /// descriptor is read. Default value is "false".
+    /// the referenced file does not exist at write time. When false, a missing file is treated
+    /// like any other fetch failure, following "blob-write-null-on-fetch-failure". Default value
+    /// is "false".
     static const char BLOB_WRITE_NULL_ON_MISSING_FILE[];
     /// "blob-write-null-on-fetch-failure" - Whether to write NULL for a descriptor BLOB value when
     /// the referenced data cannot be fetched at write time (e.g. invalid descriptor or invalid
-    /// offset). A missing file is handled by "blob-write-null-on-missing-file". When false, the
-    /// write fails when the descriptor is read. Default value is "false".
+    /// offset). A missing file is handled by "blob-write-null-on-missing-file" when that option is
+    /// enabled. When false, the write fails when the descriptor is read. Default value is "false".
     static const char BLOB_WRITE_NULL_ON_FETCH_FAILURE[];
     /// "global-index.enabled" - Whether to enable global index for scan. Default value is "true".
     static const char GLOBAL_INDEX_ENABLED[];

include/paimon/global_index/global_index_reader.h

@@ -47,7 +47,7 @@ class PAIMON_EXPORT GlobalIndexReader : public FunctionVisitor<std::shared_ptr<G
     /// @return true if the reader is thread-safe; false otherwise.
     virtual bool IsThreadSafe() const = 0;
 
-    /// @return An identifier representing the index type. (e.g., "bitmap", "lumina").
+    /// @return An identifier representing the index type. (e.g., "btree", "lumina").
     virtual std::string GetIndexType() const = 0;
 };
 

include/paimon/global_index/global_index_write_task.h

@@ -36,7 +36,7 @@ class PAIMON_EXPORT GlobalIndexWriteTask {
     ///
     /// @param table_path   Path to the table root directory where index files are stored.
     /// @param field_name   Name of the indexed column (must be present in the table schema).
-    /// @param index_type   Type of global index to build (e.g., "bitmap", "lumina").
+    /// @param index_type   Type of global index to build (e.g., "btree", "lumina").
     /// @param index_split  The indexed split containing the actual data (e.g., Parquet file) and
     //                      row id range [from, to] for data to build index.
     ///                     The range must be fully contained within the data covered