refactor: remove Lance-based data paths — HF/local data hosting only#1384
Open
Luodian wants to merge 2 commits into
Open
refactor: remove Lance-based data paths — HF/local data hosting only#1384Luodian wants to merge 2 commits into
Luodian wants to merge 2 commits into
Conversation
The public repo should not depend on Lance tables or private object-store layouts for benchmark data; Hugging Face (plus user-local files via the standard env overrides) is the only supported data host. - egotempo: drop the hardcoded S3 tar-index extraction fast path (private bucket/endpoint/Lance table + hand-parsed ~/.aws/credentials, dead code behind a silent except for anyone without that exact deployment). The HF/local resolution chain (EGOTEMPO_VIDEO_DIR / EGOTEMPO_CACHE_DIR / HF_HOME + temporal-slice dicts) is unchanged. - countix, minerva: drop the optional COUNTIX_LANCE_* / MINERVA_LANCE_* blob-resolver branches; local-dir and URL fallbacks are unchanged. - delete _task_utils/lance_video_resolver.py and the two tools/bench_minerva_* scripts whose purpose was benchmarking the Lance video pipeline. No remaining lance/pylance references in the package.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Policy
Benchmark data in the public repo resolves from Hugging Face (or user-local files via the documented env overrides) only. Lance-table-based data paths — which presume a private object-store deployment — are removed rather than parameterized (supersedes #1377).
Changes
~/.aws/credentials; it was dead code behind a silentexceptfor anyone without that exact deployment). The resolution chain users actually rely on —EGOTEMPO_VIDEO_DIR/EGOTEMPO_CACHE_DIR/HF_HOMElookup plus temporal-slice dicts — is untouched.COUNTIX_LANCE_*/MINERVA_LANCE_*blob-resolver branches; the local-dir and URL fallbacks are unchanged.lmms_eval/tasks/_task_utils/lance_video_resolver.py(no remaining importers),tools/bench_minerva_pipeline_latency.py/tools/bench_minerva_video_resolution.py(their sole purpose was benchmarking the Lance video pipeline), andtools/minerva_to_lance.py(the Lance table builder).MINERVA_VIDEO_DIR+ YouTube fallback), Lance sections dropped fromtools/README.md,--with pylance --with pyarrowdropped fromexamples/models/minerva_dummy.sh, and current-tense "Lance-backed video" claims removed from the front-page README/docs. Historical release notes underdocs/releases/keep their prose as history.Validation
python -m py_compileon all modified task utils; ruff clean on changed fileslance/pylancereferences remain inlmms_eval/,tools/,examples/,README.md,docs/README.md— remaining hits are incidental substrings ("balanced", "at a glance", the X-LANCE lab name in a benchmark homepage URL) and historicaldocs/releases/notesReview pass (2026-07-06)
Self-review caught that the first commit left the minerva Lance builder tool and several docs still advertising the removed Lance mode (README steps pointing at deleted bench tools, an example script installing
pylance). Follow-up commit finishes the removal — see the docs-scrubbed bullet and the tightened validation grep above.