feat(filesystem): load workbook worksheets as tables (#271)

- Treat XLSX and ODS workbooks as multi-table sources, loading every non-empty
  worksheet in one ingest.
- Preserve scalar worksheet selection and support JSON lists by worksheet name
  or ID.
- Apply active dlt naming, merge matching worksheet names across globs, and
  report normalized-name collisions.
- Reject plural loads for file, CSV, and blob destinations that can only represent
  one table.
- Support spreadsheet reader hints across Python 3.10+ and document the
  workbook contract.

Author: hampsterx

.github/workflows/tests.yml

@@ -104,4 +104,8 @@ jobs:
           timezoneWindows: "European Standard Time"
 
       - name: Run Docker-free unit tests
-        run: pytest -m "not integration" -k "not odbc" -p no:cacheprovider
+        # Calamine and mq-bridge load incompatible native runtimes on macOS ARM.
+        # Exercise both, but keep workbook tests in a separate interpreter process.
+        run: |
+          pytest -m "not integration" -k "not odbc" -p no:cacheprovider --ignore=tests/dlt_filesystem/format/test_spreadsheet.py
+          pytest -m "not integration" -k "not odbc" -p no:cacheprovider tests/dlt_filesystem/format/test_spreadsheet.py -n1

docs/changelog.md

@@ -2,6 +2,9 @@
 
 ## in progress
 
+- Filesystem: Treat XLSX and ODS workbooks as multi-table sources by default,
+  while preserving one-table loads through explicit worksheet selectors.
+  Thanks, @hampsterx.
 - Filesystem: Fail ingestion when a concrete source path matches no file while
   keeping unmatched glob selections valid. Thanks, @hampsterx.
 - Filesystem: Added rsync source connector. Thanks, @oferchen.

docs/supported-sources/filesystem.md

@@ -160,6 +160,8 @@ The named-hint grammar:
 - An empty value is preserved (`#sheet_name=` gives `sheet_name` = `""`);
   a reader decides whether that means "unset".
 - Duplicate keys take the last value (`#sheet_name=a&sheet_name=b` gives `b`).
+- JSON arrays select several values when the reader accepts a list, for example
+  `#sheet_name=["events","inventory"]`.
 - If any segment of the fragment is neither a `key=value` pair nor a single
   known format, the whole `#...` is treated as a literal part of the path, so a
   real `#` in a filename keeps working. Percent-encode a literal `#` as `%23`
@@ -174,6 +176,45 @@ reader requires a `#tagname=<row-tag>` hint to define the repeated element
 that represents one record / row.
 :::
 
+(workbook-tables)=
+
+## Workbook tables
+
+XLSX and ODS workbooks are multi-table sources. Without a worksheet selector,
+omniload reads every worksheet and uses each worksheet name as its destination
+table name. Empty worksheets are skipped.
+
+`--dest-table` keeps its usual `<dataset>.<table>` syntax. For a plural workbook
+load, the dataset component is used and the table component is a placeholder:
+
+```sh
+omniload ingest \
+    --source-uri 'file://data/workbook.xlsx' \
+    --dest-uri 'duckdb:///local.duckdb' \
+    --dest-table 'landing.workbook'
+```
+
+A worksheet named `Quarterly Sales` becomes `quarterly_sales` under the default
+schema naming convention. With `--schema-naming direct`, its name remains
+`Quarterly Sales`. Distinct worksheet names that resolve to the same table under
+the active convention fail before loading, with both worksheet and workbook
+names in the error.
+
+When a glob matches several workbooks, worksheets with the same original name
+merge into one destination table. Different original names that collide after
+normalization fail instead of merging silently.
+
+Select one worksheet with `#sheet_name=<name>` or `#sheet_id=<number>`. A single
+selection keeps the existing one-table behavior and uses the table component of
+`--dest-table`. Select several worksheets with a JSON array such as
+`#sheet_name=["events","inventory"]` or `#sheet_id=[1,2]`.
+
+:::{note}
+The `csv://` and `file://` destinations each produce one output file, and the
+cloud blob destinations currently address one table path. They reject a plural
+workbook load. Select one worksheet or use a dataset-capable destination.
+:::
+
 (file-format-routing)=
 
 ## File format routing

docs/supported-sources/ods.md

@@ -4,6 +4,39 @@
 
 `omniload` reads [OpenDocument spreadsheet (ODS)] files,
 used by [OpenOffice], [LibreOffice], and other spreadsheet applications.
+By default, every nonempty worksheet is loaded into its own destination table.
+
+ODS is currently supported for read operations only.
+
+## Example: load a workbook into DuckDB
+
+```sh
+omniload ingest \
+    --source-uri 'file://path/to/workbook.ods' \
+    --dest-uri   'duckdb:///local.duckdb' \
+    --dest-table 'public.workbook'
+```
+
+The first part of `--dest-table` selects the destination dataset. The second
+part is a required placeholder for a plural workbook load. Worksheet names
+replace it as the destination table names.
+
+To load one worksheet into the table named by `--dest-table`, select it by name
+or one-based number:
+
+```sh
+omniload ingest \
+    --source-uri 'file://path/to/workbook.ods#sheet_name=events' \
+    --dest-uri   'duckdb:///local.duckdb' \
+    --dest-table 'public.events'
+```
+
+Select several worksheets by name or one-based position with a JSON array:
+
+```text
+file://path/to/workbook.ods#sheet_name=["events","inventory"]
+file://path/to/workbook.ods#sheet_id=[1,2]
+```
 
 ## Where it works
 
@@ -22,24 +55,17 @@ Gzipped files are decompressed automatically.
 The whole file is read into memory and decoded at once (ODS is not a streaming
 format); a corrupt or truncated file raises rather than loading partial data.
 Map keys are expected to be strings.
+During plural loads, worksheets without data rows are skipped because dlt has
+no row from which to create a destination table. This includes header-only sheets.
 
 ## Options
 
 Options can be defined by using reader hints. The loader is using
 [polars.read_ods], please consult its documentation about all available
 parameters and their descriptions.
 
-Please note due to introspection and automatic type casting capabilities,
-the full set of parameters is only available with Python 3.14 and higher.
-
-## Example: Load ODS file into DuckDB
-
-```sh
-omniload ingest \
-    --source-uri 'file://path/to/workbook.ods#sheet_name=events' \
-    --dest-uri   'duckdb:///local.duckdb' \
-    --dest-table 'public.events'
-```
+See {ref}`Workbook tables <workbook-tables>` for naming, glob, collision, and
+destination compatibility rules shared by XLSX and ODS.
 
 
 [LibreOffice]: https://en.wikipedia.org/wiki/LibreOffice

docs/supported-sources/xlsx.md

@@ -2,10 +2,41 @@
 
 # XLSX
 
-`omniload` reads [Excel Workbook] XLSX spreadsheet files.
+`omniload` reads [Excel Workbook] XLSX spreadsheet files. By default, every
+nonempty worksheet is loaded into its own destination table.
 
 XLSX is currently supported for read operations only.
 
+## Example: load a workbook into DuckDB
+
+```sh
+omniload ingest \
+    --source-uri 'file://path/to/workbook.xlsx' \
+    --dest-uri   'duckdb:///local.duckdb' \
+    --dest-table 'public.workbook'
+```
+
+The first part of `--dest-table` selects the destination dataset. The second
+part is a required placeholder for a plural workbook load. Worksheet names
+replace it as the destination table names.
+
+To load one worksheet into the table named by `--dest-table`, select it by name
+or one-based number:
+
+```sh
+omniload ingest \
+    --source-uri 'file://path/to/workbook.xlsx#sheet_name=events' \
+    --dest-uri   'duckdb:///local.duckdb' \
+    --dest-table 'public.events'
+```
+
+Select several worksheets by name or one-based position with a JSON array:
+
+```text
+file://path/to/workbook.xlsx#sheet_name=["events","inventory"]
+file://path/to/workbook.xlsx#sheet_id=[1,2]
+```
+
 ## Where it works
 
 Excel XLSX files can be accessed on every source that goes through the shared file readers:
@@ -22,24 +53,17 @@ Gzipped files are decompressed automatically.
 The whole file is read into memory and decoded at once (XLSX is not a streaming
 format); a corrupt or truncated file raises rather than loading partial data.
 Map keys are expected to be strings.
+During plural loads, worksheets without data rows are skipped because dlt has
+no row from which to create a destination table. This includes header-only sheets.
 
 ## Options
 
 Options can be defined by using reader hints. The loader is using
 [polars.read_excel], please consult its documentation about all available
 parameters and their descriptions.
 
-Please note due to introspection and automatic type casting capabilities,
-the full set of parameters is only available with Python 3.14 and higher.
-
-## Example: Load XLSX file into DuckDB
-
-```sh
-omniload ingest \
-    --source-uri 'file://path/to/workbook.xlsx#sheet_name=events' \
-    --dest-uri   'duckdb:///local.duckdb' \
-    --dest-table 'public.events'
-```
+See {ref}`Workbook tables <workbook-tables>` for naming, glob, collision, and
+destination compatibility rules shared by XLSX and ODS.
 
 
 [Excel workbook (XLSX)]: https://en.wikipedia.org/wiki/Microsoft_Excel#Current_file_extensions

pyproject.toml

@@ -202,6 +202,7 @@ optional-dependencies.test = [
   "testcontainers[azurite,mysql,postgres]>=4.9.1,<4.16",
   "types-requests<3",
   "verlib2",
+  "xlsxwriter<4",
 ]
 urls.Changelog = "https://omniload.readthedocs.io/changelog.html"
 urls.Documentation = "https://omniload.readthedocs.io/"

src/dlt_filesystem/source/base.py

@@ -29,6 +29,29 @@ def supports_filesystem_incremental(self) -> bool:
         """Return whether the source supports file-level mtime selection."""
         return True
 
+    def produces_multiple_tables(self, uri: str, table: str) -> bool:
+        """Return whether a workbook selection dispatches worksheet tables."""
+        from dlt_filesystem.source.error import UnsupportedEndpointError
+        from dlt_filesystem.source.format.readers import (
+            spreadsheet_selection_is_plural,
+        )
+        from dlt_filesystem.source.router import (
+            blob_hints,
+            determine_endpoint,
+            parse_uri,
+        )
+
+        parsed_uri = urlparse(uri)
+        _, path = parse_uri(parsed_uri, table)
+        try:
+            endpoint = determine_endpoint(table, path)
+        except (UnsupportedEndpointError, ValueError):
+            return False
+        return endpoint in {
+            "read_excel",
+            "read_ods",
+        } and spreadsheet_selection_is_plural(blob_hints(parsed_uri, table))
+
     @staticmethod
     def endpoint_namespace(endpoint: Union[str, None], default: str) -> str:
         """

src/dlt_filesystem/source/core.py

@@ -1,4 +1,4 @@
-from typing import Union
+from typing import Any, Union
 
 import dlt
 from dlt.extract import DltResource, DltSource
@@ -42,11 +42,18 @@ def resource_for_reader(ref: FilesystemReference) -> Union[DltSource, DltResourc
 
     # Apply parameter bindings for certain readers.
     # TODO: Can this be generalized? Why not always loop in column_names into reader hints?
+    reader_kwargs: dict[str, Any] = dict(ref.hints)
+    if ref.reader_name in {"read_excel", "read_ods"}:
+        # The filesystem lister yields pages of files. Keep one collision registry
+        # bound to the reader so distinct worksheet names cannot normalize to the
+        # same table even when the workbooks occur in different pages.
+        reader_kwargs["worksheet_names"] = {}
+
     if ref.reader_name == "read_csv_headless":
         column_names = list(ref.column_types.keys()) if ref.column_types else None
-        reader = reader.bind(column_names=column_names, **ref.hints)
+        reader = reader.bind(column_names=column_names, **reader_kwargs)
     else:
-        reader = reader.bind(**ref.hints)
+        reader = reader.bind(**reader_kwargs)
 
     # Connect and propagate elements.
     return filesystem_resource | reader

src/dlt_filesystem/source/error.py

@@ -25,6 +25,25 @@ class UnsupportedEndpointError(Exception):
     pass
 
 
+class WorksheetNameCollisionError(ValueError):
+    """Two distinct worksheet names resolve to the same destination table."""
+
+    def __init__(
+        self,
+        *,
+        table_name: str,
+        first_sheet: str,
+        first_file: str,
+        second_sheet: str,
+        second_file: str,
+    ) -> None:
+        super().__init__(
+            f"Worksheet names {first_sheet!r} in {first_file!r} and "
+            f"{second_sheet!r} in {second_file!r} both resolve to destination "
+            f"table {table_name!r} under the active schema naming convention"
+        )
+
+
 class MissingDecoderError(UnsupportedEndpointError):
     """A routable format resolved to a reader whose decoder package is not installed.