Skip to content

t2m: retrain the generative model on v5 bind-referenced canonical data #858

Description

@fernandotonon

Part of epic #837. The current t2m CVAE (v4) was trained on canonical extractions whose absolute world quats mixed per-rig conventions (bind-frame conjugation, pre-#843) — the model learned a convention mush, which is why its output is a gentle average and why it emits no usable reference triple (its clips need the synthetic standing pose from PR #843's model-path fix).

Plan

  • Data: re-extract the full corpus with the calm-frame extractor (PR feat(anim): canonical clip extraction + motion-library v5 builder (t2m-v2 Slice B, #839) #843) — --dump-canonical now yields cross-rig-consistent world quats + per-clip restWorld/restDir. Rebuild the training windows from those dumps (extend scripts/prep-t2m-v4.py → v5), keeping the v4 lessons: 30fps world-frame windows with neutral starts, absolute-pose decoder, derived-local supervision, z=0 latent-dropout supervision.
  • Reference triple for model output: since training data is now convention-consistent, export a canonical restWorld/restDir alongside the model (dataset mean at the reference frame) and emit it from MotionGenerator — model clips then ride the SAME bind-referenced direction retarget as templates, retiring the synthetic-standing-pose shim.
  • Eval gate: harness parity vs the template path on walk/run/dance (joint deltas + fox renders); ship only if the model closes on the template quality bar, else it stays experimental/opt-in.
  • Bump the model files on the HF repo (motion/t2m.onnx + vocab) with backward-compatible naming or a version suffix.

Acceptance

  • Model-generated walk on the Hip Hop fox is upright, bind-referenced, and visually comparable to the template take.
  • Harness numbers recorded in the PR; templates remain the default until the gate passes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions