feat: add LanceWriteParams and lance_dataset_write_with_params (#20)

## Summary

- Adds `LanceWriteParams` (`max_rows_per_file`, `max_rows_per_group`,
`max_bytes_per_file`, `data_storage_version`, `enable_stable_row_ids`)
and `lance_dataset_write_with_params`. Pass `params = NULL` to inherit
upstream defaults.
- `lance_dataset_write` is now a thin delegator to `_with_params` with
`params = NULL`. Its FFI signature and ABI are unchanged vs
`upstream/main`.
- C++ gets a `lance::WriteParams` struct and a 5-arg
`lance::Dataset::write(...)` overload; the existing 4-arg overload
delegates to it with a default-constructed `WriteParams{}`.

## Motivation

`lance_dataset_write` uses upstream's defaults for everything.
Production callers need to tune output file/group sizes and the Lance
file format version; stable row ids are also a common opt-in. This
exposes those knobs without touching the simple-path API.

Closes #15.

## Design notes

- `data_storage_version` takes a string (`"2.0"`, `"2.1"`, `"stable"`,
`"legacy"`, ...) parsed via `LanceFileVersion::from_str`. Empty or
invalid strings → `LANCE_ERR_INVALID_ARGUMENT`.
- Numeric fields use `0` as a "keep upstream default" sentinel.
- `enable_stable_row_ids` is `bool` and so has **no default sentinel** —
whatever the caller writes is forwarded verbatim. Today upstream's
default is `false`, so a zero-init `LanceWriteParams` is a no-op;
struct-level docs in Rust and C/C++ headers call this out so a future
upstream flip won't be silent.
- Lance's `WriteParams` has more advanced fields (commit handlers,
progress callbacks, blob/base internals) intentionally not exposed —
they belong in Rust, not a thin C FFI.

## Review fixes (jja725's pass)

- **Stream consumption ordering** (#20 (comment) on `_with_params`): the
stream is now consumed via `from_raw` before validating
`uri`/`schema`/`mode`, so the documented "consumed regardless of return
code" contract holds on every error path. Mirrors what landed for plain
`lance_dataset_write` in #16.
- **`enable_stable_row_ids` semantics**: doc-only fix as suggested —
Rust struct, C header, and C++ header all call out that this field is
strictly an override.
- **`u64 → usize` truncation on 32-bit**: replaced silent `as usize`
casts with a `u64_to_usize(v, field)` helper that uses `usize::try_from`
and surfaces the offending field name + value as
`LANCE_ERR_INVALID_ARGUMENT` on overflow.

## Merge with `upstream/main`

Resolved conflicts in `Cargo.toml`, `include/lance/lance.{h,hpp}`,
`src/writer.rs`, `tests/c_api_test.rs`, and
`tests/cpp/{test_c_api.c,test_cpp_api.cpp}`. Adopted upstream's:

- `lance` 4.0.1 dependency set (added `lance-file` for
`LanceFileVersion` parsing).
- `int32_t mode` FFI signature (ABI-safe against `-fshort-enums` on the
consumer side).
- RAII `StreamGuard` / `SchemaGuard` pattern in the C++ `Dataset::write`
wrappers.
- `RwLock<Arc<Dataset>>` shape for `LanceDataset.inner`.

Diff vs `upstream/main`: 466 insertions across 5 files (`Cargo.toml`,
`include/lance/lance.h`, `include/lance/lance.hpp`, `src/writer.rs`,
`tests/c_api_test.rs`).

## Test plan

- `cargo test` — **103 passed**, 0 failed. 8 new tests for
`_with_params`: NULL params behaves like plain `write`,
`max_rows_per_file` splits fragments (100 / 20 → ≥5),
`max_rows_per_group` plumbed, `max_bytes_per_file` plumbed, known
versions (`"2.0"` / `"2.1"` / `"stable"`) accepted, empty version
rejected, invalid version rejected, `enable_stable_row_ids` toggle
accepted.
- `cargo clippy --all-targets -- -D warnings` clean.
- `cargo fmt --check` clean.
- `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps` clean.
- `cargo test --test compile_and_run_test -- --ignored` — C and C++
integration tests pass.

Author: LuciferYang

Cargo.toml

@@ -20,6 +20,7 @@ crate-type = ["cdylib", "staticlib", "rlib"]
 [dependencies]
 lance = { version = "4.0.1", features = ["substrait"] }
 lance-core = "4.0.1"
+lance-file = "4.0.1"
 lance-index = "4.0.1"
 lance-io = "4.0.1"
 lance-linalg = "4.0.1"

include/lance/lance.h

@@ -581,6 +581,48 @@ int32_t lance_dataset_write(
     LanceDataset** out_dataset
 );
 
+/**
+ * Tunable parameters for lance_dataset_write_with_params. Numeric fields
+ * default-out via 0; `data_storage_version` defaults out via NULL.
+ *
+ * Note: `enable_stable_row_ids` is a `bool` and therefore has no default
+ * sentinel — callers that zero-initialize this struct end up explicitly
+ * setting it to false (which matches upstream's current default).
+ */
+typedef struct LanceWriteParams {
+    /* Soft cap on rows per data file. 0 = default. */
+    uint64_t    max_rows_per_file;
+    /* Soft cap on rows per row group. 0 = default. */
+    uint64_t    max_rows_per_group;
+    /* Soft cap on bytes per data file (~90 GB upstream default). 0 = default. */
+    uint64_t    max_bytes_per_file;
+    /* Lance file format version, e.g. "2.0", "2.1", "stable", "legacy".
+     * NULL = default. Invalid strings → LANCE_ERR_INVALID_ARGUMENT. */
+    const char* data_storage_version;
+    /* Opt into stable row ids (better for compaction at a small write cost).
+     * Strictly an override — see struct-level note above. */
+    bool        enable_stable_row_ids;
+} LanceWriteParams;
+
+/**
+ * Same as lance_dataset_write but takes a LanceWriteParams for tuning the
+ * output shape. Pass `params` = NULL to use defaults (equivalent to calling
+ * lance_dataset_write directly).
+ *
+ * @return 0 on success, -1 on error. See lance_dataset_write for the error
+ *         code list; invalid `data_storage_version` also returns
+ *         LANCE_ERR_INVALID_ARGUMENT.
+ */
+int32_t lance_dataset_write_with_params(
+    const char* uri,
+    const struct ArrowSchema* schema,
+    struct ArrowArrayStream* stream,
+    int32_t mode,
+    const LanceWriteParams* params,
+    const char* const* storage_opts,
+    LanceDataset** out_dataset
+);
+
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

include/lance/lance.hpp

@@ -19,6 +19,7 @@
 
 #include <cstdint>
 #include <memory>
+#include <optional>
 #include <stdexcept>
 #include <string>
 #include <utility>
@@ -103,6 +104,22 @@ enum class WriteMode : int32_t {
     Overwrite = LANCE_WRITE_OVERWRITE,
 };
 
+/// Tunable parameters for Dataset::write. Numeric fields default-out via 0;
+/// `data_storage_version` defaults out via `std::nullopt`.
+///
+/// `enable_stable_row_ids` has no default sentinel — whatever value the
+/// caller writes is forwarded to upstream. Today this matches upstream's
+/// default (`false`), so a default-constructed WriteParams is a no-op; if
+/// upstream ever changes its default, callers must set this field explicitly.
+struct WriteParams {
+    uint64_t                   max_rows_per_file    = 0;
+    uint64_t                   max_rows_per_group   = 0;
+    uint64_t                   max_bytes_per_file   = 0;
+    /// Lance file format version, e.g. "2.0", "2.1", "stable", "legacy".
+    std::optional<std::string> data_storage_version;
+    bool                       enable_stable_row_ids = false;
+};
+
 // ─── Dataset ─────────────────────────────────────────────────────────────────
 
 class Dataset {
@@ -145,14 +162,26 @@ class Dataset {
         WriteMode mode,
         const std::vector<std::pair<std::string, std::string>>& storage_opts = {}) {
 
+        return write(uri, stream, mode, WriteParams{}, storage_opts);
+    }
+
+    /// Same as the four-argument `write` but tunes the output via `params`.
+    /// Pass a default-constructed `WriteParams{}` to inherit upstream defaults.
+    static Dataset write(
+        const std::string& uri,
+        ArrowArrayStream* stream,
+        WriteMode mode,
+        const WriteParams& params,
+        const std::vector<std::pair<std::string, std::string>>& storage_opts = {}) {
+
         if (stream == nullptr) {
             throw Error(LANCE_ERR_INVALID_ARGUMENT, "stream must not be null");
         }
 
-        // RAII guard for the stream. Until `lance_dataset_write` is called,
-        // any exception (failed `get_schema`, `std::bad_alloc` while building
-        // `kv`, etc.) must release the stream. After that call Rust owns it,
-        // so we `disarm()` immediately before invoking the C API.
+        // RAII guard for the stream. Until `lance_dataset_write_with_params`
+        // is called, any exception (failed `get_schema`, `std::bad_alloc`
+        // while building `kv`, etc.) must release the stream. After that call
+        // Rust owns it, so we `disarm()` immediately before invoking the C API.
         struct StreamGuard {
             ArrowArrayStream* s;
             bool armed = true;
@@ -219,17 +248,26 @@ class Dataset {
         const char* const* opts_ptr =
             storage_opts.empty() ? nullptr : kv.data();
 
+        LanceWriteParams c_params = {};
+        c_params.max_rows_per_file    = params.max_rows_per_file;
+        c_params.max_rows_per_group   = params.max_rows_per_group;
+        c_params.max_bytes_per_file   = params.max_bytes_per_file;
+        c_params.data_storage_version =
+            params.data_storage_version ? params.data_storage_version->c_str() : nullptr;
+        c_params.enable_stable_row_ids = params.enable_stable_row_ids;
+
         // The C API consumes the stream on every return path, so disarm the
         // guard before calling. After this point the stream pointer is logically
         // owned by Rust and any C++-side exception must not re-release it.
         stream_guard.disarm();
 
         LanceDataset* out = nullptr;
-        int32_t rc = lance_dataset_write(
+        int32_t rc = lance_dataset_write_with_params(
             uri.c_str(),
             &schema,
             stream,
             static_cast<int32_t>(mode),
+            &c_params,
             opts_ptr,
             &out);
         if (rc != 0) check_error();
@@ -240,7 +278,7 @@ class Dataset {
         // thread-local code is `LANCE_OK` on this path (rc == 0).
         if (!out) {
             throw Error(LANCE_ERR_INTERNAL,
-                        "lance_dataset_write returned success with null out_dataset");
+                        "lance_dataset_write_with_params returned success with null out_dataset");
         }
         return Dataset(out);
     }

src/writer.rs

@@ -8,6 +8,7 @@
 //! dataset with a committed manifest rather than uncommitted fragment files.
 
 use std::ffi::c_char;
+use std::str::FromStr;
 use std::sync::{Arc, RwLock};
 
 use arrow::ffi::FFI_ArrowSchema;
@@ -17,6 +18,7 @@ use arrow_schema::Schema as ArrowSchema;
 use lance::Dataset;
 use lance::dataset::{WriteMode as LanceWriteModeUpstream, WriteParams};
 use lance_core::Result;
+use lance_file::version::LanceFileVersion;
 use lance_io::object_store::{ObjectStoreParams, StorageOptionsAccessor};
 
 use crate::dataset::LanceDataset;
@@ -74,6 +76,36 @@ impl From<LanceWriteMode> for LanceWriteModeUpstream {
     }
 }
 
+/// Tunable parameters for `lance_dataset_write_with_params`.
+///
+/// Numeric fields use `0` as a "keep upstream default" sentinel, and
+/// `data_storage_version` uses NULL the same way. The struct is `#[repr(C)]`
+/// and ABI-stable within a minor version.
+///
+/// `enable_stable_row_ids` is a `bool` and therefore has **no default
+/// sentinel** — whatever the caller writes is forwarded verbatim to upstream.
+/// Callers that zero-initialize this struct (the documented way to inherit
+/// other defaults) end up explicitly setting `enable_stable_row_ids = false`.
+/// This matches upstream's current default, so the behavior is identical
+/// today; if upstream ever changes that default, callers must set this field
+/// explicitly to follow.
+#[repr(C)]
+pub struct LanceWriteParams {
+    /// Soft cap on rows per data file. `0` uses upstream's default.
+    pub max_rows_per_file: u64,
+    /// Soft cap on rows per row group. `0` uses upstream's default.
+    pub max_rows_per_group: u64,
+    /// Soft cap on bytes per data file (~90 GB by default). `0` uses upstream's default.
+    pub max_bytes_per_file: u64,
+    /// Lance file format version string, e.g. `"2.0"`, `"2.1"`, `"stable"`,
+    /// `"legacy"`. NULL uses upstream's default. Invalid strings are rejected
+    /// with `LANCE_ERR_INVALID_ARGUMENT`.
+    pub data_storage_version: *const c_char,
+    /// Strictly an override (no default sentinel — see struct-level docs).
+    /// Whatever value the caller writes is forwarded to upstream.
+    pub enable_stable_row_ids: bool,
+}
+
 /// Write an Arrow record batch stream to a Lance dataset at `uri`, committing a manifest.
 ///
 /// - `uri`: Dataset URI (`file://`, `s3://`, `memory://`, ...). Must not be NULL or empty.
@@ -88,6 +120,8 @@ impl From<LanceWriteMode> for LanceWriteModeUpstream {
 ///   version on success (caller closes). Pass NULL to discard. On error
 ///   `*out_dataset` is untouched — do not read or free it.
 ///
+/// Equivalent to `lance_dataset_write_with_params(..., params = NULL, ...)`.
+///
 /// Returns 0 on success, -1 on error.
 #[unsafe(no_mangle)]
 pub unsafe extern "C" fn lance_dataset_write(
@@ -97,9 +131,36 @@ pub unsafe extern "C" fn lance_dataset_write(
     mode: i32,
     storage_opts: *const *const c_char,
     out_dataset: *mut *mut LanceDataset,
+) -> i32 {
+    unsafe {
+        lance_dataset_write_with_params(
+            uri,
+            schema,
+            stream,
+            mode,
+            std::ptr::null(),
+            storage_opts,
+            out_dataset,
+        )
+    }
+}
+
+/// Same as `lance_dataset_write` but takes a `LanceWriteParams` for tuning the
+/// output shape. Pass `params = NULL` to use upstream defaults.
+#[unsafe(no_mangle)]
+pub unsafe extern "C" fn lance_dataset_write_with_params(
+    uri: *const c_char,
+    schema: *const FFI_ArrowSchema,
+    stream: *mut FFI_ArrowArrayStream,
+    mode: i32,
+    params: *const LanceWriteParams,
+    storage_opts: *const *const c_char,
+    out_dataset: *mut *mut LanceDataset,
 ) -> i32 {
     ffi_try!(
-        unsafe { write_dataset_inner(uri, schema, stream, mode, storage_opts, out_dataset) },
+        unsafe {
+            write_dataset_inner(uri, schema, stream, mode, params, storage_opts, out_dataset)
+        },
         neg
     )
 }
@@ -109,6 +170,7 @@ unsafe fn write_dataset_inner(
     schema: *const FFI_ArrowSchema,
     stream: *mut FFI_ArrowArrayStream,
     mode: i32,
+    params: *const LanceWriteParams,
     storage_opts: *const *const c_char,
     out_dataset: *mut *mut LanceDataset,
 ) -> Result<i32> {
@@ -198,13 +260,20 @@ unsafe fn write_dataset_inner(
         storage_options_accessor: Some(Arc::new(StorageOptionsAccessor::with_static_options(opts))),
         ..ObjectStoreParams::default()
     });
-    let params = WriteParams {
+    let mut write_params = WriteParams {
         mode: mode.into(),
         store_params,
         ..WriteParams::default()
     };
+    if !params.is_null() {
+        // SAFETY: `params` is non-NULL (checked above) and the caller
+        // guarantees it points to a properly-initialized `LanceWriteParams`
+        // valid for the duration of this call. We borrow by shared reference
+        // and only read; the borrow is consumed before any await point.
+        unsafe { apply_write_params(&mut write_params, &*params)? };
+    }
 
-    let dataset = block_on(Dataset::write(reader, uri_str, Some(params)))?;
+    let dataset = block_on(Dataset::write(reader, uri_str, Some(write_params)))?;
 
     if !out_dataset.is_null() {
         let handle = LanceDataset {
@@ -221,3 +290,53 @@ unsafe fn write_dataset_inner(
 
     Ok(0)
 }
+
+/// Apply caller-provided overrides onto an `lance::WriteParams`. Zero / NULL
+/// fields are no-ops so upstream defaults flow through.
+unsafe fn apply_write_params(target: &mut WriteParams, params: &LanceWriteParams) -> Result<()> {
+    if params.max_rows_per_file > 0 {
+        target.max_rows_per_file = u64_to_usize(params.max_rows_per_file, "max_rows_per_file")?;
+    }
+    if params.max_rows_per_group > 0 {
+        target.max_rows_per_group = u64_to_usize(params.max_rows_per_group, "max_rows_per_group")?;
+    }
+    if params.max_bytes_per_file > 0 {
+        target.max_bytes_per_file = u64_to_usize(params.max_bytes_per_file, "max_bytes_per_file")?;
+    }
+    if !params.data_storage_version.is_null() {
+        // SAFETY: `data_storage_version` is non-NULL (checked above) and the
+        // `apply_write_params` caller (`unsafe fn` contract) guarantees it
+        // points to a NUL-terminated C string valid for the duration of the
+        // outer FFI call. `parse_c_string` reads by shared reference and the
+        // returned borrow is consumed before this function returns.
+        //
+        // `parse_c_string` returns `None` only for NULL input, which the
+        // outer check already ruled out. `.filter` lets an empty C string
+        // also fail presence, producing the clearer message below instead
+        // of relying on `FromStr`'s generic "unknown version" path.
+        let s = unsafe { helpers::parse_c_string(params.data_storage_version)? }
+            .filter(|s| !s.is_empty())
+            .ok_or_else(|| lance_core::Error::InvalidInput {
+                source: "data_storage_version must not be an empty string".into(),
+                location: snafu::location!(),
+            })?;
+        let version =
+            LanceFileVersion::from_str(s).map_err(|e| lance_core::Error::InvalidInput {
+                source: format!("invalid data_storage_version {s:?}: {e}").into(),
+                location: snafu::location!(),
+            })?;
+        target.data_storage_version = Some(version);
+    }
+    target.enable_stable_row_ids = params.enable_stable_row_ids;
+    Ok(())
+}
+
+/// Narrow `u64 -> usize` with an explicit error on overflow (32-bit targets).
+/// Realistic write tunings fit in `usize` on every supported target, but a
+/// silent `as` cast would wrap on a 32-bit host.
+fn u64_to_usize(v: u64, field: &'static str) -> Result<usize> {
+    usize::try_from(v).map_err(|_| lance_core::Error::InvalidInput {
+        source: format!("{field}={v} exceeds usize::MAX on this target").into(),
+        location: snafu::location!(),
+    })
+}