docs: document multi-worker DataLoader support for remote tables

[mintlify]

Summary

Document that remote LanceDB tables can now be used directly with multi-worker PyTorch DataLoaders, and refresh guidance on the optional with_connection_factory escape hatch.

Changes

• Add a "Using multiple DataLoader workers" section to the PyTorch integration page covering num_workers, spawn, and persistent_workers.
• Add a subsection showing remote (db://) tables working in worker processes out of the box.
• Add a subsection showing how to provide a custom connection factory when credentials should be loaded inside the worker.

Context

Triggered by an upstream change that lets remote tables carry their connection state through pickling so they reopen correctly in PyTorch DataLoader workers, while keeping the connection factory available for custom credential loading.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lancedb-bcbb4faf	🟢 Ready	View Preview	Jun 1, 2026, 9:57 AM

[prrao87]

cc @westonpace - I presume these docs make sense based on your latest additions.

[prrao87]

I pushed a follow-up commit to make the PyTorch examples match current LanceDB behavior.

What I tested:

• Built the latest lancedb main locally from /Users/prrao/code/lancedb with maturin develop.
• Installed torch==2.12.1.
• Ran python/tests/test_torch.py: 9 passed, 2 skipped.
• Added temporary docs-level repros in /private/tmp/test_lancedb_torch_docs_snippets.py: 4 passed.
• Reproduced the direct DataLoader(table, ...) example without collate_fn; it fails with PyTorch default collation:

TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'pyarrow.lib.ChunkedArray'>

Why this change is necessary:

• Direct LanceDB Table datasets emit Arrow data, and PyTorch's default collate_fn does not accept Arrow batches directly.
• The direct Table examples need collate_fn=tbl_to_tensor to produce tensors.
• Permutation examples work as written with PyTorch's default collation because the default Permutation output is a list of Python dicts, which PyTorch batches into a dict of tensors.

[prrao87]

@westonpace could you please review this PyTorch docs update when possible? As I was testing the code that the agent wrote, some more recent changes that we merged recently caused this error to surface.

TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'pyarrow.lib.ChunkedArray'>

I've patched a fix that uses tbl_to_tensor from the utils that were added, and it works now. Just want to be sure the explanation in the PyTorch page makes sense.

Also, are we missing anything in the documentation about how to handle these on RemoteTable specifically?

[westonpace]

Some suggestions but nice examples

[westonpace]

Hmm, Permutation can also support direct Arrow-to-tensor conversion. It just isn't the default. This makes it sound like you'd have to use a Table.

[prrao87]

Yup, makes sense! Fixed.

[westonpace]

Actually forkserver is probably better than spawn (and will be the new python default). I'd say that should be our preference.

Once the streaming dataset is available my guidance would be to use forkserver and use num_workers=1 unless you can prove you have GIL contention in your trasform function.

[prrao87]

Done. Switched the preference to forkserver (and added a note that it's POSIX-only, so Windows falls back to spawn).

On num_workers=1: since that guidance is gated on the streaming dataset, and that's still WIP / held until release (per #294), I've left the examples at num_workers=4 for now so this page doesn't document unshipped behavior. Tracking the num_workers=1 recommendation to land alongside the streaming dataset docs in #294 when it lands on main.