t2m v5: flow-matching motion model — canonical training data + in-graph sampler + reference triple (#840, #858)#888
Conversation
…pler, reference triple (#840, #858) Data (#858): scripts/prep-t2m-v5.py canonicalizes every clip (corpus *.canonical.json dumps + CMU BVH) into a rig-independent representation before windowing — per joint, aim-from-canonical-T-pose composed with the swing/twist residual: Q'(f) = twist(theta, d(f)) * arc(D_c -> d(f)). Identical for every rig performing the same motion, and exactly what the bind-referenced retarget consumes given the canonical reference triple (restWorld = identity, restDir = D_c). 17,286 windows / 18 actions (corpus 517 + CMU 16,789), replacing the v4 per-rig convention mush. Model (#840): scripts/train-t2m-flow-v5.py — small DiT-style velocity transformer (7.3M params, 6D rotations, AdaLN-Zero conditioning) trained with the rectified-flow objective + classifier-free guidance; flow matching samples a transport path to a single data mode instead of decoding a conditional mean (the CVAE's structural weakness). The Euler sampler (+CFG) is UNROLLED INSIDE the exported ONNX graph, so the shipped MotionGenerator contract (tokens[1,V], seed[1,Z] -> motion [1,T,220]) is unchanged — seed is the flattened noise tensor. Wiring (#858): MotionGenerator parses the vocab's restWorld/restDir into the generated clip; CLI/MCP/GUI model paths pass it to applyMotionClip, so model clips ride the SAME bind-referenced direction retarget as v5 template clips — the synthetic-standing-pose shim remains only as the legacy-v4 fallback. applyMotionClip now treats any valid unit-norm reference quat as present (identity counts; only zero-filled entries mean absent), which the v5 identity triple requires. Render-verified on the Rumba rig: model walk/run/wave/dance all upright and temporally coherent end-to-end (the v4 CVAE folded or tumbled); templates remain the default + automatic fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c2589b033
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| huggingface-cli upload "$REPO" "$OUT_DIR/t2m.onnx" motion/t2m.onnx \ | ||
| --commit-message "t2m v5: flow-matching model, canonical reference triple (#840/#858)" | ||
| huggingface-cli upload "$REPO" "$OUT_DIR/t2m-vocab.json" motion/t2m-vocab.json \ | ||
| --commit-message "t2m v5: vocab + canonical reference triple (#840/#858)" |
There was a problem hiding this comment.
Uploading the v5 artifacts over the same motion/t2m.onnx and motion/t2m-vocab.json paths leaves existing opt-in users on the old v4 model: MotionGenerator::ensureModelBlocking() returns immediately when both cached files already exist, so clients that previously downloaded v4 never fetch the new vocab with restWorld/restDir and never exercise the v5 retarget path. Use a versioned cache filename or add a vocab/version check that invalidates the old local files before relying on this upload to roll out v5.
Useful? React with 👍 / 👎.
|
Superseded by the combined PR #904 (all three stacked slices merged into one branch, |
1 participant
Closes #840, closes #858 (epic #837). Stacked on #886 (base =
feat/t2m-twist-857, the twist transport makes the canonicalization lossless) — retarget tomasterafter #886 merges.The three pieces
1. Canonical training data (#858) —
scripts/prep-t2m-v5.pyre-extracts the full corpus into a rig-independent representation before windowing. Per joint, using each clip's own reference triple (the same one the bind-referenced retarget consumes), the bone's direction trajectory and swing/twist roll residual are rebuilt against a FIXED canonical T-pose direction set:Q' is identical for every rig performing the same motion — the v4 "per-rig convention mush" (the measured cause of the CVAE's gentle-average output) is gone by construction. 17,286 windows / 18 actions (corpus 517 + CMU 16,789; the v4 cache had 47 CMU clips' worth). The neutral-start gate is deliberately dropped: model clips now ride the direction retarget, which references the reference triple, not clip frame 0 — any phase is a valid window (huge data win).
2. Flow-matching model (#840) —
scripts/train-t2m-flow-v5.py: 7.3M-param DiT-style velocity transformer (6D rotations, AdaLN-Zero conditioning), rectified-flow objective + classifier-free guidance, class-balanced sampling (walk = 64% of windows). Flow matching samples a transport path to a single mode of the data instead of decoding a conditional mean — the architectural fix for the CVAE's structural weakness (per the epic). The Euler sampler (+CFG) is unrolled inside the exported ONNX graph, so the shippedMotionGeneratorcontract is byte-identical:tokens[1,V], seed[1,Z] → motion[1,T,220]withseed= the flattened noise tensor (Z = T·132). No C++ sampler loop needed; best-of-N candidate scoring still works unchanged.3. Reference triple wiring (#858) — the vocab json ships the model's canonical triple (
restWorld= identity ×22,restDir= the canonical T-pose directions).MotionGeneratorparses it into the generated clip; the CLI/MCP/GUI model paths pass it toapplyMotionClip, so model clips ride the same bind-referenced direction retarget as v5 template clips — the synthetic-standing-pose shim is retired to a legacy-v4 fallback.applyMotionClip's restWorld presence check is now norm-based (identity counts as present; only zero-filled entries mean absent) — the old non-identity check would have silently demoted v5 model clips.Evaluation (honest)
Render-verified on the Rumba rig (isometric protocol): model walk / run / wave / dance are all upright and temporally coherent end-to-end — every v4 CVAE variant folded, tumbled, or collapsed late-clip. Walk and run read as their actions; wave/dance are coherent but gentler than the template takes.
Gate verdict: templates remain the default and automatic fallback (per the epic's eval gate — the model ships opt-in behind
--model/model:true/ the GUI checkbox). This is still a step-change over the v4 CVAE for the opt-in path.Hosting:
scripts/upload-t2m-v5-model.shpreserves the v4 files under versioned names, then uploads the v5 model+vocab. Backward-compatible: the ONNX interface is unchanged, and older builds ignore the vocab's extra keys (they just keep using the standing-pose shim).🤖 Generated with Claude Code