Epic: Text-to-motion v2 — permissive motion corpus + flow-matching model

[fernandotonon]

Successor to #411 and the #793 decision record: no licensable pretrained text-to-motion model exists (HumanML3D/AMASS models are non-commercial; HY-Motion 1.0 is territory-restricted incl. its Outputs and derivatives-by-distillation), and our from-scratch CVAE's weakness is structural — a z=0 conditional-mean decoder averages modes into gentle motion (measured in the v4 training notes). Nobody in the open ecosystem has assembled a permissive motion corpus at scale — so we build one, then train the right architecture on it.

Why this can work

• Data: thousands of permissively-licensed animated humanoid rigs exist across Sketchfab (API filters: CC0/CC-BY + animated + downloadable), OpenGameArt, and CC0 asset packs (Quaternius, Kenney, KayKit, Poly Pizza) — plus the full CMU corpus (~2,600 trials; we currently mine 47 clips) and SFU mocap (free incl. commercial, attribution). Asset names/tags provide free text labels ("zombie walk", "sword slash") — far richer than the current 13-action vocabulary.
• Infrastructure we already own: the 22-joint canonical skeleton + retarget (applyMotionClip, canonicalIndexForBone), library windowing/energy selection (build-motion-library.py), a C++ flow-matching sampler loop (TripoSG), and ONNX inference everywhere.
• Architecture: flow-matching/diffusion doesn't mode-average — it is the single change that separates MDM-era quality from our CVAE, independent of data scale.

Slices

• A — corpus scraper (scripts/scrape-motion-corpus.py, offline dev tool): license-filtered discovery + download (Sketchfab Data API cc0/by, OpenGameArt, curated CC0 packs), per-asset provenance manifest (source URL, author, license, tags), ATTRIBUTION.md generation, validation via qtmesh info --json (skeleton + animations present). CC0/CC-BY only; Mixamo, LAFAN1, Bandai-Namco, AMASS and game rips explicitly excluded.
• B — canonical normalization + template library v5: retarget every scraped clip onto the 22-joint canonical skeleton (rig→canonical, the reverse of applyMotionClip; BVH path exists for CMU), active-window selection, keyword extraction from tags → a much larger clip library (hundreds of actions, several takes each). Immediate quality win with zero ML risk — the template path is the current quality bar.
• C — flow-matching motion DiT: small transformer trained on the corpus (CMU + scraped) with tag-based text conditioning; export to ONNX; C++ Euler flow loop at inference (TripoSG pattern). Replaces the CVAE as the --model path; template library stays default + fallback.
• D — evaluation + wiring: fixed prompt set, side-by-side vs template clips, latency budget on M-series CPU, THIRD_PARTY_AI_MODELS.md provenance section.

Non-goals

• Matching 1B-param model fidelity — target is "hundreds of real actions, keyword-matched, with a generative model that interpolates between them" for game-dev blocking animation.
• Distillation from restricted models (see the Evaluate HY-Motion 1.0 as the text-to-motion backend (ONNX export + quantization) #793 record: HY-Motion's license makes distilled models Derivatives).

🤖 Generated with Claude Code