feat(scripts): license-filtered motion corpus scraper (t2m-v2 Slice A, #838)#841
Conversation
…lice A) Offline dev tool assembling a permissively-licensed animated-humanoid corpus for the text-to-motion v2 epic (#837): - Sketchfab Data API discovery (animated + downloadable), client-side license filter by exact label/slug (CC0 / CC Attribution only — the search API omits slugs and "CC Attribution" must not prefix-match the NonCommercial/ShareAlike/NoDerivs variants); downloads via SKETCHFAB_API_TOKEN, discovery-only rows recorded without it. - OpenGameArt scraping with the per-page License(s) block as the authoritative filter (every listed license must be CC0/CC-BY), 3D-Art check, attachment-only downloads with zip extraction. - Quaternius CC0 packs enumerated live via the GitHub tree API of the mirror repo (hardcoded pack URLs rot — fetch-training-rigs.sh lesson), Patreon-Exclusive folders excluded, one manifest row per pack. - Validation with the editor's own loader: `qtmesh info --json` keeps only assets with a skeleton and >= 1 skeletal animation (--keep-static opts out). - Incremental provenance manifest (merged across runs, deduped by source_url) + generated ATTRIBUTION.md for CC-BY credits. Live-tested: 16 assets (15 Quaternius packs + 1 CC-BY OGA rig with 46 bones / 11 animations); validation correctly dropped signs, posed statues and unanimated base meshes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 55 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9204914c97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| info = json.loads(out.stdout or "{}") | ||
| except Exception: | ||
| continue | ||
| skel = info.get("skeleton") or {} |
There was a problem hiding this comment.
When qtmesh info --json loads a file with multiple entities, cmdInfo emits a JSON array, so json.loads returns a list and this info.get(...) raises AttributeError outside the try block. Many downloaded GLTF/FBX character archives contain more than one mesh entity, so a single such asset aborts the whole validation run instead of selecting the best skeletal entry or skipping that file.
Useful? React with 👍 / 👎.
| if not adir: | ||
| kept.append(a) # discovery-only entries stay listed |
There was a problem hiding this comment.
This path treats every asset without _dir as discovery-only, but Sketchfab download failures also have no _dir and are still appended with download: failed. On a rerun where a previously validated Sketchfab asset hits a transient download/API error, write_outputs sees the failed row as fresh and replaces the prior manifest entry with an entry that has no files or validation; skip failed downloads or keep them out of the replacement set.
Useful? React with 👍 / 👎.
| print(f" pack: {pack} ({len(files)} file(s))") | ||
| assets.append({ | ||
| "source": "packs", | ||
| "source_url": "https://quaternius.com/packs/", |
There was a problem hiding this comment.
Every Quaternius pack is recorded with the same source_url, while the incremental merge later dedupes/replaces assets solely by source_url. If a later --packs run partially fails or changes --max-per-query, the presence of any current pack removes all prior pack rows sharing this URL, losing provenance for packs that were not successfully refreshed; use a pack-specific URL or another stable unique key.
Useful? React with 👍 / 👎.
|
1 participant
Slice A of the text-to-motion v2 epic #837: an offline dev tool that assembles a permissively-licensed animated-humanoid corpus for the Slice B library upgrade and Slice C flow-matching model.
Sources (CC0/CC-BY only — Mixamo/LAFAN1/Bandai-Namco/AMASS/game-rips excluded by policy, same bar as
fetch-training-rigs.sh):SKETCHFAB_API_TOKEN; discovery-only rows without it.Validation dogfoods the editor: every file goes through
qtmesh info --json; only assets with a skeleton and ≥1 skeletal animation are kept.Provenance: incremental
manifest.json(merged across runs, deduped by source URL) + generatedATTRIBUTION.mdfor CC-BY credits — both must ship with anything derived from the corpus.Live-tested: 16 assets in one corpus (15 Quaternius animated packs — men/women/robot/zombie/animals, up to 14 animations each — plus a CC-BY OGA mannequin with 46 bones/11 animations); validation correctly dropped signs, posed statues, and unanimated base meshes. Sketchfab discovery verified against real CC-BY listings.
Part of #837; implements #838.
🤖 Generated with Claude Code