Add EgoDex hand-pose scenario search example (datasets/egodex)#44
Open
sgarimel wants to merge 17 commits into
Open
Add EgoDex hand-pose scenario search example (datasets/egodex)#44sgarimel wants to merge 17 commits into
sgarimel wants to merge 17 commits into
Conversation
Read Apple's EgoDex through Daft's LeRobot v3 reader, precompute per-frame geometric states + Daft-window action rates and SigLIP-2 semantic embeddings, and serve a query UI that combines a geometric filter with semantic ranking. Dataset is CC-BY-NC-ND: scripts contain no EgoDex data; users convert their own licensed copy locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tures Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Avoids colliding with the real daft.datasets.hdf5 reader shipping in Daft soon, and makes the conversion example self-contained (imports the local copy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…df5File Add the thin facade the blog/notebook import (convert -> add_state_features -> add_skeleton_features -> embed_frames -> query) and a copy-paste egodex_demo.py showing pose-only, text-only, and combined pose x text queries. Switch egodex_lerobot.py to read HDF5 via Daft's native Hdf5File type (one row per episode) instead of the vendored hdf5_example reader. Add requirements-egodex.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sgarimel
force-pushed
the
shreyas/lerobot-pose-pipeline
branch
from
a4bdbd6 to
80cc10b
Compare
… precompute Drop the NumPy grip predicates (is_writing_grip / is_hammer_grip / calibrate_grip_thresholds) from skeleton_features.py: grip detection now lives only in the Daft layer (egodex._is_writing_grip / _is_hammer_grip as @daft.func, with thresholds from egodex.calibrate at query time). skeleton_features stays pure NumPy geometry. Collapse run_pose_features.py onto the facade (read -> add_state_features -> add_skeleton_features -> write_parquet); it now writes continuous geometry only, no baked sc_* booleans. Refactor query_ui.py to source its predicate/thresholds/segments/text-encoder from the facade and read the continuous-only features, so the whole repo computes scenarios at query time from one implementation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build -> query walkthrough: read, add_state_features, add_skeleton_features, embed_frames, write; then calibrate, pose query, semantic text query, overlay, and launch the query_ui dashboard. Saved with cleared outputs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pass numpy frames directly to the SigLIP processor, run embeddings before pose features in the demo, and bump daft/gradio deps. Co-authored-by: Cursor <cursoragent@cursor.com>
Consolidate facade and feature modules under egodex_lib/, drop vendored readers and standalone run scripts in favor of PEP 723 demos, and update the README and query UI for the new layout.
… cleanup Switch HDF5 conversion to process_transforms via daft.File, add notebook download skip logic with .data caching, use nightly Daft in PEP 723 demos, and drop egodex_lib/__init__ in favor of direct submodule imports.
Replace the manual batch loop with from_files → process_egodex_episode → LeRobotDatasetWriter, fix the episode struct return dtype, and restore output-dir cleanup before writing.
…entual-Inc/daft-examples into shreyas/lerobot-pose-pipeline we need this# the commit.
Move HDF5 read + feature build into the writer UDF, switch write_lerobot to from_glob_path over .data, and tidy the demo notebook/script.
Switch the demo to read raw EgoDex HDF5 directly via read_egodex (native Hdf5File) instead of a LeRobot round-trip, and fix issues found while running it end to end: - _decode_sibling_mp4: strip Daft's file:// URI scheme before av.open - SiglipEmbedder.embed_image: guard empty batches (HF processor does images[0].device and errors on []) - overlay: glob with recursive=True so `**` matches nested dirs and episode_index ordering matches read_egodex Rewrite README for the egodex_lib/ layout and the raw-HDF5 pipeline, with a note on Daft laziness and the explode barrier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eline Reconcile with Everett's EgoDexLeRobotDatasetWriter commit. The two branches took different directions in convert_egodex_to_lerobot.py (LeRobot-writer vs raw-HDF5 read_egodex); keep the read_egodex pipeline, notebook, demo.py, and README from this branch. Everett's writer refactor stays reachable in history, and his egodex_video.py formatting is preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
datasets/egodex/— a Daft example that finds specific physical states and actions in Apple's EgoDex egocentric hand-manipulation video ("a writing-gripped hand", "the wrist twisting", "fingers closing into a grasp") that text-only search can't reach.Pipeline:
egodex_lerobot.py— convert raw EgoDex HDF5 → LeRobot v3 (48-D state + 204-D skeleton + video)run_pose_features.py— per-frame geometric states (NumPy) + action rates via Daft window functions → parquetrun_clip_features.py— SigLIP-2 image embeddings via a batched@daft.clsGPU UDF → parquetquery_ui.py— Gradio demo: filter by pose scenario, rank by semantic text, play matched segmentsReads through the vendored LeRobot v3 reader (
lerobot.py), one row per frame with lazy video decode. Users bring their own EgoDex copy and run the conversion locally.Notes for review
./egodex_lerobot_full.hdf5_example.pyis a vendored HDF5 reader (the realdaft.datasets.hdf5ships in Daft soon).query_ui.pypinsgradio>=5,<6(Gradio 6 stopped executing the inline<script>the player relies on).🤖 Generated with Claude Code