docs: clarify build and commit integration

Author: zjw1111

docs/source/build_system.rst

@@ -48,11 +48,27 @@ for data format and file system.
 
    project(MyExample)
 
-   find_package(Arrow REQUIRED)
-   find_package(Paimon REQUIRED)
+   # Arrow's static package may export LZ4 under a different target name.
+   find_package(lz4 CONFIG QUIET)
+   if(TARGET LZ4::lz4_static AND NOT TARGET LZ4::lz4)
+     add_library(LZ4::lz4 ALIAS LZ4::lz4_static)
+   elseif(TARGET lz4::lz4 AND NOT TARGET LZ4::lz4)
+     add_library(LZ4::lz4 ALIAS lz4::lz4)
+   endif()
+
+   find_package(Arrow CONFIG REQUIRED)
+   find_package(Paimon CONFIG REQUIRED)
+
+   if(TARGET Arrow::arrow_shared)
+     set(PAIMON_ARROW_TARGET Arrow::arrow_shared)
+   elseif(TARGET Arrow::arrow_static)
+     set(PAIMON_ARROW_TARGET Arrow::arrow_static)
+   else()
+     message(FATAL_ERROR "No supported Arrow CMake target is available")
+   endif()
 
    add_executable(my_example my_example.cc)
-   target_link_libraries(my_example PRIVATE Arrow::arrow_shared
+   target_link_libraries(my_example PRIVATE ${PAIMON_ARROW_TARGET}
                                             Paimon::paimon_shared
                                             Paimon::paimon_parquet_file_format_shared
                                             Paimon::paimon_local_file_system_shared)
@@ -64,12 +80,15 @@ The directive ``find_package(Paimon REQUIRED)`` instructs CMake to locate a
 Paimon C++ installation on your system. If successful, it sets ``Paimon_FOUND``
 to true if the Paimon C++ libraries were found.
 
-It also defines the following linkable imported targets:
+It defines the following supported imported target:
 
 * ``Paimon::paimon_shared`` links to the Paimon shared libraries
-* ``Paimon::paimon_static`` links to the Paimon static libraries
 
-In most cases, it is recommended to use the Paimon shared libraries.
+Static targets such as ``Paimon::paimon_static`` may also be present when
+Paimon is built with ``PAIMON_BUILD_STATIC=ON``. They are not currently
+supported as standalone installed targets because their exported link
+interfaces do not include every third-party dependency. Installed consumers
+should use the shared targets.
 
 Optional plugins (built-in file formats, file systems, and index)
 -----------------------------------------------------------------
@@ -78,22 +97,24 @@ Paimon provides a set of built-in optional plugins that you can link to as neede
 
 - File format plugins:
 
-  - ``Paimon::paimon_parquet_file_format_shared`` / ``Paimon::paimon_parquet_file_format_static``
-  - ``Paimon::paimon_orc_file_format_shared`` / ``Paimon::paimon_orc_file_format_static``
-  - ``Paimon::paimon_avro_file_format_shared`` / ``Paimon::paimon_avro_file_format_static``
-  - ``Paimon::paimon_blob_file_format_shared`` / ``Paimon::paimon_blob_file_format_static``
+  - ``Paimon::paimon_parquet_file_format_shared``
+  - ``Paimon::paimon_orc_file_format_shared``
+  - ``Paimon::paimon_avro_file_format_shared``
+  - ``Paimon::paimon_blob_file_format_shared``
 
 - File system plugins:
 
-  - ``Paimon::paimon_local_file_system_shared`` / ``Paimon::paimon_local_file_system_static``
-  - ``Paimon::paimon_jindo_file_system_shared`` / ``Paimon::paimon_jindo_file_system_static``
+  - ``Paimon::paimon_local_file_system_shared``
+  - ``Paimon::paimon_jindo_file_system_shared``
 
 - Index plugins:
 
-  - ``Paimon::paimon_file_index_shared`` / ``Paimon::paimon_file_index_static``
-  - ``Paimon::paimon_lumina_index_shared`` / ``Paimon::paimon_lumina_index_static``
-  - ``Paimon::paimon_lucene_index_shared`` / ``Paimon::paimon_lucene_index_static``
+  - ``Paimon::paimon_file_index_shared``
+  - ``Paimon::paimon_lumina_index_shared``
+  - ``Paimon::paimon_lucene_index_shared``
 
 .. note::
 
-  In most cases, it is recommended to use the shared variants of these plugins.
+  Static plugin targets have the same installed-consumer limitation as
+  ``Paimon::paimon_static`` and should not be used as standalone imported
+  targets.

docs/source/building.rst

@@ -159,14 +159,22 @@ require project-specific patches, so their supported source values are
      -Dzstd_SOURCE=BUNDLED
 
 Use ``PAIMON_PACKAGE_PREFIX`` to provide one common prefix for dependencies
-whose own ``<Package>_ROOT`` variable is not set.
+whose own ``<Package>_ROOT`` variable is not set. Because the patched Arrow and
+ORC dependencies cannot be resolved from the system, a global ``SYSTEM`` build
+must override them to ``BUNDLED``:
 
 .. code-block:: shell
 
    cmake -B build \
      -DPAIMON_DEPENDENCY_SOURCE=SYSTEM \
+     -DArrow_SOURCE=BUNDLED \
+     -DORC_SOURCE=BUNDLED \
      -DPAIMON_PACKAGE_PREFIX=/opt/paimon-deps
 
+All other enabled dependencies must be available as system packages or under
+the specified prefix. When ORC support is disabled, the ``ORC_SOURCE`` override
+can be omitted.
+
 Package-manager-specific modes are intentionally out of scope for this first
 dependency source interface. They can still be used through standard CMake
 mechanisms such as ``CMAKE_PREFIX_PATH`` or ``CMAKE_TOOLCHAIN_FILE``, while

docs/source/user_guide/write.rst

@@ -17,10 +17,10 @@
 
 Write
 =====
-Batch writing requires the compute engine to pre-bucket data (bucket), using the
-same bucketing strategy as Paimon to ensure correct ``Scan`` behavior, and to
-specify the target ``partition``. Data should be accumulated into ``RecordBatch``
-and written to Paimon.
+Batch writing requires the compute engine to specify the target ``partition``.
+For fixed-bucket tables, the engine must also assign a valid bucket to every
+``RecordBatch``. In unaware-bucket and postpone-bucket modes, the writer can
+resolve the bucket automatically when it is omitted.
 
 Paimon C++ uses Apache Arrow as the :ref:`in-memory columnar format<memory-format>`
 to more efficiently support writing to disk columnar formats such as ORC,
@@ -33,7 +33,6 @@ Parquet, and Avro, thereby improving write throughput.
 
   Not supported in the current scope:
     - Changelog
-    - Indexes
 
 Bucketing Modes
 ---------------
@@ -56,9 +55,15 @@ RecordBatch Construction
 
 - The compute engine must:
 
-  - Apply the Paimon-consistent bucketing function to each row prior to batching.
   - Assign the correct ``partition`` for each row.
-  - Group rows into Arrow ``RecordBatch`` per partition-bucket combination to minimize writer state changes and I/O overhead.
+  - In fixed-bucket mode, apply the Paimon-consistent bucketing function, set a
+    bucket in ``[0, bucket)``, and group rows into Arrow ``RecordBatch`` objects
+    per partition-bucket combination.
+
+- In unaware-bucket mode (append table with ``bucket = -1``), an omitted bucket
+  is resolved to ``0``.
+- In postpone-bucket mode (primary-key table with ``bucket = -2``), an omitted
+  bucket is resolved to ``-2``.
 
 - Recommended practices:
 
@@ -122,26 +127,27 @@ produce a correct ``Snapshot``, which commonly includes (but is not limited to):
 
 .. note::
 
-   Current C++ scope supports Append and PK tables. Changelog is out of
-   scope and should not be emitted in ``CommitMessage`` until
-   explicitly supported.
+   The C++ writer supports Append and PK tables and can produce
+   ``CommitMessage`` objects for both. ``FileStoreCommit`` currently executes
+   local commits only for append-only tables on non-object-store file systems.
+   PK and object-store commit messages must be sent to an external control
+   plane. Changelog is out of scope and should not be emitted in
+   ``CommitMessage`` until explicitly supported.
 
 Serialization and Deserialization
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- Binary Format:
-  - The binary payload must strictly conform to Java Paimon’s ``CommitMessage`` encoding.
-  - Version tags or schema identifiers should be included to enable forwards/backwards compatibility and safe upgrades.
-
-- Serialization API:
-  - Provide a function to serialize the writer’s commit state into a byte buffer (or stream) consumable by Java Paimon.
-
-- Deserialization API:
-  - Provide a function to parse a Java-produced ``CommitMessage`` binary payload back into C++ commit structures for verification, replay, and testing.
-
-- Validation:
-  - Include conformance tests to assert that C++ serialized payloads are accepted by Java Paimon.
-  - Include round-trip tests to ensure C++ can parse Java-produced payloads and vice versa for supported message versions.
+- **Binary format:** The binary payload must strictly conform to Java Paimon's
+  ``CommitMessage`` encoding. It does not contain a version tag, so callers
+  must transport ``CommitMessage::CurrentVersion()`` separately and supply it
+  when deserializing.
+- **Serialization API:** Use ``CommitMessage::Serialize`` for one message or
+  ``CommitMessage::SerializeList`` for a list.
+- **Deserialization API:** Use ``CommitMessage::Deserialize`` or
+  ``CommitMessage::DeserializeList`` with the separately supplied
+  serialization version.
+- **Validation:** Conformance and round-trip tests must verify compatibility
+  with Java Paimon for supported message versions.
 
 Operational Flow
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -155,12 +161,15 @@ Operational Flow
 
 3. Each writer invokes ``PrepareCommit``, which:
    - Aggregates per-writer state into a ``CommitMessage``.
-   - Serializes the message into a Java-compatible binary payload.
+   - Returns ``CommitMessage`` objects; it does not serialize them.
 
-4. The compute engine gathers ``CommitMessages`` from all writers.
+4. The compute engine gathers ``CommitMessage`` objects from all writers. For
+   cross-process transport, it explicitly calls ``Serialize`` or
+   ``SerializeList`` and carries ``CurrentVersion()`` alongside the payload.
 
-5. The compute engine issues a ``Commit`` request to the control plane with the
-   collected messages, resulting in a new ``Snapshot``.
+5. For a supported local append-table commit, the engine passes the objects to
+   ``FileStoreCommit``. For PK tables or object-store paths, it sends the
+   serialized payload and version to an external control plane.
 
-6. The coordinator validates the messages, updates manifests/metadata, and
-   finalizes the snapshot atomically.
+6. The local committer or external coordinator validates the messages, updates
+   manifests/metadata, and finalizes the snapshot atomically.

examples/CMakeLists.txt

@@ -20,23 +20,40 @@ project(example)
 
 set(CMAKE_CXX_STANDARD 17)
 
+# Arrow's static export refers to LZ4::lz4, while some lz4 packages export a
+# differently named target. Define a compatibility alias before loading Arrow.
+find_package(lz4 CONFIG QUIET)
+if(TARGET LZ4::lz4_static AND NOT TARGET LZ4::lz4)
+    add_library(LZ4::lz4 ALIAS LZ4::lz4_static)
+elseif(TARGET lz4::lz4 AND NOT TARGET LZ4::lz4)
+    add_library(LZ4::lz4 ALIAS lz4::lz4)
+endif()
+
 find_package(Arrow CONFIG REQUIRED)
 find_package(Paimon CONFIG REQUIRED)
 
+if(TARGET Arrow::arrow_shared)
+    set(PAIMON_EXAMPLE_ARROW_TARGET Arrow::arrow_shared)
+elseif(TARGET Arrow::arrow_static)
+    set(PAIMON_EXAMPLE_ARROW_TARGET Arrow::arrow_static)
+else()
+    message(FATAL_ERROR "Neither Arrow::arrow_shared nor Arrow::arrow_static is available")
+endif()
+
 add_executable(read_write_demo read_write_demo.cpp)
 add_executable(clean_demo clean_demo.cpp)
 
 set(CMAKE_EXE_LINKER_FLAGS "-Wl,--no-as-needed ${CMAKE_EXE_LINKER_FLAGS}")
 
 target_link_libraries(read_write_demo
-                      PRIVATE Arrow::arrow_shared
+                      PRIVATE ${PAIMON_EXAMPLE_ARROW_TARGET}
                               Paimon::paimon_shared
                               Paimon::paimon_parquet_file_format_shared
                               Paimon::paimon_orc_file_format_shared
                               Paimon::paimon_local_file_system_shared)
 
 target_link_libraries(clean_demo
-                      PRIVATE Arrow::arrow_shared
+                      PRIVATE ${PAIMON_EXAMPLE_ARROW_TARGET}
                               Paimon::paimon_shared
                               Paimon::paimon_parquet_file_format_shared
                               Paimon::paimon_orc_file_format_shared

include/paimon/commit_message.h

@@ -32,8 +32,12 @@ namespace paimon {
 class CommitMessageSerializer;
 class MemoryPool;
 
-/// Commit message for partition and bucket. Support Serialize and Deserialize, compatible with java
-/// version.
+/// Commit message for partition and bucket. Supports serialization and deserialization compatible
+/// with the Java version.
+///
+/// @note Serialized payloads do not embed their serialization version. Transport
+/// `CurrentVersion()` alongside the payload and pass it explicitly to `Deserialize()` or
+/// `DeserializeList()`.
 // TODO(yonghao.fyh): to add some statistics of write (e.g., write bytes)
 class PAIMON_EXPORT CommitMessage {
  public:
@@ -44,13 +48,15 @@ class PAIMON_EXPORT CommitMessage {
 
     /// Serializes a single commit message to a binary string format.
     /// The serialized format is compatible with the Java version of Paimon.
+    /// The serialization version is not included in the returned payload.
     /// @param commit_message The commit message to serialize.
     /// @param pool Memory pool for memory allocation during serialization.
     /// @return Result containing the serialized string data, or an error if serialization fails.
     static Result<std::string> Serialize(const std::shared_ptr<CommitMessage>& commit_message,
                                          const std::shared_ptr<MemoryPool>& pool);
 
     /// Serializes a list of commit messages to a binary string format.
+    /// The serialization version is not included in the returned payload.
     /// @param commit_messages Vector of commit messages to serialize.
     /// @param pool Memory pool for memory allocation during serialization.
     /// @return Result containing the serialized string data, or an error if serialization fails.

include/paimon/file_store_commit.h

@@ -43,6 +43,11 @@ class CommitMessage;
 ///
 /// The `FileStoreCommit` class provides interfaces for committing changes, expiring old snapshots,
 /// dropping partitions, and retrieving commit metrics.
+///
+/// @note Local commit execution currently supports append-only tables on non-object-store file
+/// systems. `Create()` returns `NotImplemented` for primary-key tables and object-store paths.
+/// Primary-key writers can still produce `CommitMessage` objects; those messages must be committed
+/// by an external control plane.
 class PAIMON_EXPORT FileStoreCommit {
  public:
     /// Create an instance of `FileStoreCommit`.