feat(sql): load remote SQLite and DuckDB databases

[hampsterx]

Summary

• Load SQLite and DuckDB databases that live on a remote filesystem, addressed by the object URI the storage service already names them with: s3://analytics/snapshots/events.duckdb.
• Route a filesystem-scheme source URI whose object carries a database extension to the SQL source rather than a format reader, so --source-table keeps selecting a table inside the database.
• Resolve the engine from the staged file's header, so .db needs no disambiguation and a mislabeled object still loads.
• Stage the object for the complete ingestion lifetime with size checks, byte verification, cleanup, and no remote writeback.

Changes

• Add object staging to dlt_filesystem and database routing to the SQL source, covering s3://, r2://, gs://, az://, adls://, and abfss:// through the filesystem credential parsing those schemes already share.
• Identify the engine from SQLite's header marker or DuckDB's DUCK magic, falling back to an unambiguous extension only for an empty object.
• Reject selections a single database connection cannot open: globs, # fragments, credentials in the authority, and the split form that puts the object path on --source-table.
• Validate a dry run against the SQL source the real run uses, without downloading the object.
• Exclude mq-bridge-py==0.3.9 after its newly published macOS build corrupted values and failed the MQBridge suite; clean resolution selects 0.3.8.
• Document the contract on filesystem.md, keep the DuckDB and SQLite pages usage-first, and record the grammar plus rejected alternatives in ADR-001.

Test plan

• Full Docker-free suite: 980 passed, 44 skipped
• S3, Azure, and GCS emulator cases: 10 passed in tests/main/filesystem/test_remote_integration.py
• Ruff formatting/checks, ty, metadata validation
• Strict Sphinx build
• Independent Codex and GLM-5.2 reviews, with confirmed findings fixed or dispositioned
• GitHub Actions unit matrix on macOS and Windows, Linux integration matrix, Codecov, Read the Docs, gitleaks, and OCI build

References

• Database: Load file-based databases from remote filesystems #226

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e9ca2dc5-2310-4d66-b570-ed840dc708a0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[read-the-docs-community]

Documentation build overview

📚 omniload | 🛠️ Build #33881079 | 📁 Comparing d944285 against latest (747b6aa)

  🔍 Preview build  

8 files changed · + 1 added · ± 7 modified

+ Added

• decisions/ADR-001-stage-remote-file-databases.html

± Modified

• changelog.html
• index.html
• supported-sources/duckdb.html
• supported-sources/filesystem.html
• supported-sources/ods.html
• supported-sources/sqlite.html
• supported-sources/xlsx.html

[codecov]

Codecov Report

❌ Patch coverage is 95.49550% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.15%. Comparing base (7e3d5f8) to head (d944285).

Files with missing lines	Patch %	Lines
src/dlt_filesystem/staging.py	94.38%	5 Missing ⚠️
src/dlt_filesystem/util/auth.py	91.52%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #272      +/-   ##
==========================================
+ Coverage   59.45%   60.15%   +0.70%     
==========================================
  Files         234      236       +2     
  Lines       10977    11142     +165     
==========================================
+ Hits         6526     6703     +177     
+ Misses       4451     4439      -12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amotl]

Thank you for this improvement. I've added a comment with a question if the routing could be made differently, so that s3://analytics/snapshots/events.duckdb could be used as an input URL as-is.

[amotl]

Is it possible to treat .duckdb and .sqlite resources using a reader, or otherwise dispatch processing differently, so that ingesting from s3://analytics/snapshots/events.duckdb actually becomes possible without encoding that request in a different way? The URL reads so well!

[hampsterx]

Yes, and it reads better this way. Pushed: s3://analytics/snapshots/events.duckdb now works as written, and the location= parameter form is gone.

Routing. A filesystem-scheme source URI whose object carries a database extension (.db, .ddb, .duckdb, .sqlite, .sqlite3) goes to the SQL source instead of a format reader. --source-table main.events keeps selecting a table inside the database, and SQL reflection, chunking, and type mapping stay on the path. Credentials and storage options are the same query parameters the matching file source takes, so there is no second encoding layer:

omniload ingest \
  --source-uri 's3://analytics/snapshots/events.duckdb?access_key_id=ACCESS&secret_access_key=SECRET' \
  --source-table 'main.events' \
  --dest-uri 'duckdb:///local.duckdb' \
  --dest-table 'raw.events'

I went with dispatch rather than a reader. A reader would have to reimplement reflection, chunking, type mapping and table selection that the SQL source already owns, one engine at a time, and dlt_filesystem would grow a SQLAlchemy dependency. The routing decision stays in omniload; dlt_filesystem only gained a generic "stage this object" primitive.

Engine detection is from the file header, SQLite's marker at offset 0 and DuckDB's DUCK behind its checksum, because .db names both engines and an object's name is not evidence of its contents. So a mislabeled object still loads, and a non-database reports that rather than failing somewhere inside the driver. Only an empty object has no header, and it falls back to an unambiguous extension so a freshly created database stays loadable.

Scope: s3://, r2://, gs://, az://, adls://, abfss://, the schemes whose credential parsing is already shared with the file sources. The connectors added in GH-254 build their fsspec client inside dlt_source(), so staging cannot reach them without lifting that construction into a shared "authorized filesystem for this URI" helper. Happy to do that as a follow-up if you want databases on the other transports; it is the same seam GH-256 is about.

Two things the change deliberately does not do. The split form (--source-uri s3:// plus the object path on --source-table) is rejected with a message naming the carrier to use, since the table has to name a table inside the database. And a dry run now validates against the SQL source the real run uses, instead of the storage source, without downloading the object.

Docs: the contract lives in one place on filesystem.md (Database files), with the DuckDB and SQLite pages carrying just the example and a link. ADR-001 records the grammar and the rejected alternatives.

Verified: full Docker-free suite (980 passed, 44 skipped), the S3, Azure and GCS emulator cases (10 passed in that file), ruff, ty, and a strict Sphinx build. Independent Codex and GLM-5.2 reviews both flagged the split form, which is now handled; GLM caught the dry-run inconsistency.

[amotl]

That sounds excellent, thank you. Now that the other PR got merged first, this one only needs to resolve conflicts and will be ready to go.