Add streaming, bounded, credential-re-vending HF ingest CLI

[justinrmiller]

Why

A one-shot tbl.add(staged) of a multi-TB dataset on a db:// enterprise connection dies after ~1h with S3 ExpiredToken. The naive flow points one Arrow dataset at the whole repo and writes it in a single Lance operation:

staged = ds.dataset("~/datacomp_meta", format="parquet")  # ~3TB metadata
tbl.add(staged)   # one long write outlives the vended STS token -> ExpiredToken

The table is opened once, the vended STS credentials are baked into the Lance backend, and that single write can't finish inside the token's lifetime.

What

A new ingest-hf-streaming CLI that makes the client-only mitigation copy-pasteable, matching the existing ingest-images style. Two independent levers, both in geneva_examples/core/utils/hf_streaming.py:

1. Bounded sub-writes — iter_hf_batches streams the source into --chunk-rows-sized RecordBatches and appends one per chunk, so no single Lance write outlives the STS TTL.
2. Per-chunk re-vend — fresh_table returns a table handle with freshly vended credentials before each append. Each chunk logs the vended aws_session_token prefix so a real db:// run self-verifies that credentials rotate.

Source modes:

• datasets (default) — load_dataset(streaming=True) buffered into batches; works for any datasets-streamable repo with Arrow-serializable columns.
• parquet — streams parquet straight from hf:// via HfFileSystem + pyarrow.dataset (exact schema, no decode, no full download); scales to datacomp-style sharded metadata.

--resume reads count_rows() and skips already-written source rows, making a failed multi-TB load idempotent-ish on restart.

The re-vend lever (design note)

The intuitive lever — re-open the table with conn.open_table() to re-vend — could not be confirmed by static analysis. conn.open_table() does build a fresh Table/_ltbl, but the actual STS vend happens inside opaque native (inner.open_table), and tbl.add() writes straight to the cached _ltbl without refreshing. geneva's own credential-refresh primitive is latest_storage_options() (invoked only in get_reference() for remote workers, never on the local write path). The only lever guaranteed to re-vend by construction is reconstructing the connection.

So --revend-mode is pluggable, defaulting to the guaranteed option:

Value	Behavior
connect (default)	Reconstruct geneva.connect per chunk → fresh namespace client → fresh describe_table vend. Guaranteed to re-vend.
reopen	Reuse the connection, re-open the table per chunk. Lighter; re-vend happens in opaque native code and is unconfirmed.
latest	Hold one table, refresh in place via latest_storage_options() (private/internal lancedb API).

Files

• New geneva_examples/core/utils/hf_streaming.py — iter_hf_batches, fresh_table, vended_token_prefix, and the network-bound source readers (# pragma: no cover).
• New geneva_examples/pipeline/ingest_hf_streaming.py — the CLI.
• New tests/test_hf_streaming.py — 21 tests (no network/cluster).
• pyproject.toml — ingest-hf-streaming script entry + coverage omit (like the other cluster CLIs).
• README.md — "Streaming ingest" section with before/after, examples, and a full options reference.
• .gitignore — ignore huggingface_cache/ (pre-existing gap that ingest-images shares via HF_HOME).

Verification

• make check green: ruff lint + format, 131 passed, 98.99% coverage; hf_streaming.py at 100%.
• Datasets-mode local run (local-dir connection): streamed cornell-movie-review-data/rotten_tomatoes, wrote 3 chunks (20+20+10 = exactly --limit 50), table_rows 50; observed a distinct LanceNamespaceDBConnection per chunk under connect mode, confirming the re-vend fires. token=<none> as expected on a local dir.
• Parquet reader verified in isolation (hf:// glob → schema → to_batches, ~1.4s).

Still needs a real db:// cluster (couldn't run here)

• The re-vend assertion: confirm the token= prefix rotates across chunks under --revend-mode connect (and check whether reopen keeps a stable token — that would empirically confirm connect must remain the default).
• A final enterprise smoke run with a modest --limit.

🤖 Generated with Claude Code