Skip to content

feat: add namespace insert write path#587

Open
jackye1995 wants to merge 2 commits into
lance-format:mainfrom
jackye1995:jack/namespace-insert-writes
Open

feat: add namespace insert write path#587
jackye1995 wants to merge 2 commits into
lance-format:mainfrom
jackye1995:jack/namespace-insert-writes

Conversation

@jackye1995
Copy link
Copy Markdown
Contributor

@jackye1995 jackye1995 commented Jun 6, 2026

Adds an opt-in namespace insert append path for namespace-backed Lance tables.

Expected user experience:

  • Existing INSERT INTO and append writes keep using the default Spark writer unless users set use_namespace_insert=true.
  • With namespace insert enabled, users still submit a normal DataFrame append; Spark runs executor-side writer tasks that send Arrow batches through the Lance namespace insert API.
  • namespace_insert_parallelism lets users request the number of writer tasks; sharded tables use the sharding distribution, and unsharded tables repartition by the first output column.
  • Directory and REST namespaces use the same API path, so REST namespaces can handle ingestion without changing Spark user code.
  • Insert requests commit as they run, so users should use the default writer when they need Spark driver-side atomic commit behavior.

Includes local/rest-dir Docker CI coverage and documentation updates.

@github-actions github-actions Bot added the enhancement New feature or request label Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant