Skip to content

Commit 465fb6a

Browse files
geng-haorangeng-haoran
committed
docs(maniskill): comprehensive final integration page (15 tasks, all layouts, reward/success/demo/assets)
Update the integration docs to the landed state: 15 shipped native tasks across all robot layouts (panda / panda_stick / multi-robot), action-level 1:1, dense reward 10/10 + success 12/12 bitwise, is_grasped, demo .h5 replay, per-episode reset distribution, HF-backed clone-deletable assets, backward compatibility, and the remaining per-episode internal-geometry follow-up.
1 parent b6eff1a commit 465fb6a

1 file changed

Lines changed: 92 additions & 70 deletions

File tree

‎docs/source/dataset_benchmark/integrations/maniskill.md‎

Lines changed: 92 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,76 @@
11
# ManiSkill ↔ MetaSim / SAPIEN3 integration (native 1:1)
22

3-
[ManiSkill3](https://github.com/haosulab/ManiSkill) is a SAPIEN-backed
4-
manipulation benchmark. RoboVerse ships **MetaSim-native** ManiSkill
5-
tabletop tasks that reproduce the native ManiSkill (`physx_cpu`) rollout
6-
**1:1** through the standard `BaseTaskEnv` + SAPIEN3 handler path — no
7-
runtime `mani_skill` import, so the clone is deletable.
3+
[ManiSkill3](https://github.com/haosulab/ManiSkill) is a SAPIEN-backed manipulation
4+
benchmark. RoboVerse ships **MetaSim-native** ManiSkill tasks that reproduce the
5+
native ManiSkill (`physx_cpu`) rollout **1:1** through the standard `BaseTaskEnv` +
6+
SAPIEN3 handler path — no runtime `mani_skill` import, so the clone is deletable.
87

98
Live report (videos + measured parity):
109
<http://localhost:8000/#roboverse/maniskill_integration>.
1110

1211
## What "1:1" means here
1312

14-
Native ManiSkill on `sim_backend="physx_cpu"` is bit-deterministic, so a
15-
clean SAPIEN3 scene can reproduce it exactly. The reproduction recipe
16-
(all opt-in `SimParamCfg` knobs on the SAPIEN3 handler, default-off so
17-
existing tasks are byte-identical):
18-
19-
1. **Disable gravity on every robot link** (`sapien_disable_robot_gravity`)
20-
— how ManiSkill holds the arm; the single biggest dynamics factor.
21-
2. **Full PhysX config set globally** before scene creation
22-
(`sapien_apply_global_physx`): solver iters, PCM/TGS, contact/rest
23-
offset, sleep/bounce thresholds, default material — mirrors
24-
`BaseEnv._set_scene_config`.
13+
Native ManiSkill on `sim_backend="physx_cpu"` is bit-deterministic, so a clean SAPIEN3
14+
scene can reproduce it exactly. The reproduction recipe is a set of **opt-in**
15+
`SimParamCfg` knobs on the SAPIEN3 handler (all default-off, so existing tasks are
16+
byte-identical):
17+
18+
1. **Disable gravity on every robot link** (`sapien_disable_robot_gravity`) — how
19+
ManiSkill holds the arm; the single biggest dynamics factor.
20+
2. **Full PhysX config set globally** before scene creation (`sapien_apply_global_physx`):
21+
solver iters, PCM/TGS, contact/rest offset, sleep/bounce thresholds, default
22+
material — mirrors `BaseEnv._set_scene_config`.
2523
3. **Drive** with `force_limit` + `mode="force"` (`sapien_drive_force_mode`).
26-
4. **Table** as a kinematic box with the ground plane far below
27-
(`sapien_ground_altitude`); `PrimitiveCubeCfg`/`PrimitiveMultiBoxCfg`
28-
honor `fix_base_link` (kinematic).
29-
5. **Controller** = ManiSkill `pd_joint_delta_pos` + gripper mimic
30-
(vendored in `_native/control.py`), decimation `sim_freq // control_freq`.
31-
32-
## Shipped tasks
33-
34-
`import roboverse_pack.tasks.maniskill` registers 12 tabletop tasks as
35-
`maniskill.<name>_native` (also `<name>_native`):
36-
37-
| Task | object pose Δ vs native | dense reward | success |
38-
| --- | --- | --- | --- |
39-
| `pick_cube` | 4.7e-6 | bitwise (5.96e-8) | ✓ |
40-
| `push_cube` | 2.3e-7 | bitwise | ✓ |
41-
| `pull_cube` | 2.3e-7 | bitwise | ✓ |
42-
| `stack_cube` | 1.5e-6 | bitwise | ✓ |
43-
| `poke_cube` | 2.9e-7 | — | ✓ |
44-
| `lift_peg_upright` | 3.0e-7 | bitwise | ✓ |
45-
| `roll_ball` | 1.2e-7 | bitwise | ✓ |
46-
| `place_sphere` | 1.2e-7 | bitwise | ✓ |
47-
| `stack_pyramid` | 8.4e-7 | (no native dense) | ✓ |
48-
| `pull_cube_tool` | 2.8e-7 | bitwise | ✓ |
49-
| `peg_insertion_side` | 4.8e-5 | (todo) | proxy |
50-
| `plug_charger` | 5.4e-7 | (no native dense) | proxy |
51-
52-
Object pose tracks native to PhysX float32 roundoff (1.2e-7–4.8e-5 over
53-
aggressive random steps; ~1e-6 under demo-like motion). Dense rewards
54-
match native `compute_dense_reward` to float32 epsilon (8/10 tasks with a
55-
native dense reward done); `is_grasped` matches `Panda.is_grasping`
56-
(18/18, contact forces ~0.01 N) via the new sapien3
57-
`get_pairwise_contact_force`. Reset uses ManiSkill's spawn + goal
58-
distribution.
24+
4. **Table** as a kinematic box with the ground plane far below (`sapien_ground_altitude`);
25+
`PrimitiveCubeCfg` / `PrimitiveMultiBoxCfg` honor `fix_base_link` (kinematic).
26+
5. **Controller** = ManiSkill `pd_joint_delta_pos` (`_native/control.py`,
27+
parametric over arm DOF + optional mimic gripper), decimation `sim_freq // control_freq`.
28+
29+
## Shipped tasks (15)
30+
31+
`import roboverse_pack.tasks.maniskill` registers them as `maniskill.<name>_native`:
32+
33+
| Task | robot | object pose Δ vs native | dense reward | success |
34+
| --- | --- | --- | --- | --- |
35+
| `pick_cube` | panda | 4.7e-6 | bitwise (5.96e-8) | bitwise |
36+
| `push_cube` | panda | 2.3e-7 | bitwise | bitwise |
37+
| `pull_cube` | panda | 2.3e-7 | bitwise | bitwise |
38+
| `stack_cube` | panda | 1.5e-6 | bitwise | bitwise |
39+
| `poke_cube` | panda | 2.9e-7 | bitwise | bitwise |
40+
| `lift_peg_upright` | panda | 3.0e-7 | bitwise | bitwise |
41+
| `roll_ball` | panda | 1.2e-7 | bitwise | bitwise |
42+
| `place_sphere` | panda | 1.2e-7 | bitwise | bitwise |
43+
| `stack_pyramid` | panda | 8.4e-7 | (no native dense) | bitwise |
44+
| `pull_cube_tool` | panda | 2.8e-7 | bitwise | bitwise |
45+
| `peg_insertion_side` | panda | 4.8e-5 | bitwise | bitwise |
46+
| `plug_charger` | panda | 5.4e-7 | (no native dense) | bitwise |
47+
| `push_t` | panda_stick | Tee Δ=0 | — | proxy |
48+
| `draw_triangle` | panda_stick | 9.9e-6 | — | proxy |
49+
| `two_robot_pick_cube` | 2× panda_wristcam | cube 3.3e-7 | — | proxy |
50+
51+
- **Dynamics** track native to PhysX float32 roundoff (object pose 1.2e-7–4.8e-5 over
52+
aggressive random steps; ~1e-6 under demo-like motion).
53+
- **Action-level 1:1 for every ManiSkill robot layout**: panda (7 arm + 1 mimic gripper,
54+
8-dim), panda_stick (7-dim arm-only, PushT/DrawTriangle), and multi-agent
55+
(TwoRobotPickCube, 2 × 8-dim split per robot by `ManiSkillMultiRobotTask`).
56+
- **Dense rewards**: all 10 tasks with a native dense reward match `compute_dense_reward`
57+
to float32 epsilon (5.96e-8–1.19e-7).
58+
- **Success**: all 12 tabletop predicates ported bitwise (including peg-in-hole and
59+
charger `_compute_distance`).
60+
- **`is_grasped`** matches `Panda.is_grasping` (18/18, contact forces ~0.01 N) via the new
61+
sapien3 `get_pairwise_contact_force`.
62+
- **Reset distribution** matches ManiSkill's per-episode spawn/goal sampling (a persistent
63+
RNG advances across resets; an explicit seed is reproducible).
64+
65+
## Demo replay
66+
67+
Official ManiSkill `pd_joint_delta_pos` demos replay through the shipped tasks
68+
(`tools/maniskill_integration/replay_demo.py`): seeding each episode's initial state +
69+
goal and replaying the recorded actions reproduces the demonstrated success — **21/25
70+
PickCube demos** (native `physx_cpu` itself reproduces ~24/25 open-loop; the gap is the
71+
~1e-4 CPU-handler residual near the 0.025 m boundary). Action replay is deterministic: an
72+
identical grasp+lift sequence keeps the cube within ~1e-4 m of native through the
73+
contact-rich phase.
5974

6075
## Run
6176

@@ -72,38 +87,45 @@ env = cls(sc); env.reset(seed=0)
7287
for _ in range(50): obs, rew, term, trunc, info = env.step(torch.zeros((1,8)))
7388
"
7489

75-
# measure native<->recipe parity for any task
90+
# measure native<->recipe parity (single-agent: 14 tasks; multi-agent: separate tool)
7691
SAPIEN_HEADLESS=1 python -m tools.maniskill_integration.parity_native --all --steps 30
92+
SAPIEN_HEADLESS=1 python -m tools.maniskill_integration.parity_multi_agent --task TwoRobotPickCube-v1
93+
94+
# side-by-side 1:1 video (native | shipped task | diff)
95+
SAPIEN_HEADLESS=1 python -m tools.maniskill_integration.render_parity --task PickCube-v1 --shipped pick_cube
7796

78-
# render a side-by-side 1:1 video (native | shipped task | diff)
79-
SAPIEN_HEADLESS=1 python -m tools.maniskill_integration.render_parity \
80-
--task PickCube-v1 --shipped pick_cube --steps 60
97+
# replay official ManiSkill demos through the shipped task
98+
SAPIEN_HEADLESS=1 python -m tools.maniskill_integration.replay_demo --task pick_cube --episodes 25
8199

82100
# regression tests
83-
python -m pytest tests/test_maniskill_native_task.py \
84-
tests/test_maniskill_reward_grasp.py tests/test_maniskill_success.py
101+
python -m pytest tests/test_maniskill_native_task.py tests/test_maniskill_reward_grasp.py \
102+
tests/test_maniskill_success.py tests/test_maniskill_action_levels.py \
103+
tests/test_maniskill_reset.py tests/test_maniskill_demo_replay.py
85104
```
86105

87106
## Assets
88107

89-
The ManiSkill Panda (`panda_v2.urdf` + `franka_description` meshes) is
90-
vendored under `roboverse_data/robots/maniskill_panda/` (HuggingFace-backed,
91-
like the other integrations); the locator (`_native/recipe.panda_urdf_path`)
92-
prefers the vendored copy and falls back to the installed `mani_skill`
93-
package.
108+
The ManiSkill panda assets (`panda_v2.urdf` for the gripper arm, `panda_stick.urdf`,
109+
`panda_v3.urdf` for the wrist-cam arm, + `franka_description` / realsense meshes) are
110+
vendored under `roboverse_data/robots/maniskill_panda/` and published to the
111+
HuggingFace-backed `RoboVerseOrg/roboverse_data` dataset. The locators
112+
(`_native/recipe.panda_urdf_path` etc.) prefer the local copy, else token-free
113+
`snapshot_download` from HF, else fall back to an installed `mani_skill` package — so the
114+
clone loads its robots **without** a `mani_skill` install.
94115

95116
## Backward compatibility
96117

97-
All MetaSim-side changes are opt-in (`SimParamCfg` knobs default to
98-
`None`/`False`; the new `PrimitiveMultiBoxCfg` and
99-
`get_pairwise_contact_force` are additive) — existing SAPIEN3 tasks are
100-
byte-identical (417 sapien3 + general MetaSim tests pass). The RoboVerse
101-
side is purely additive (new `_native/` package + tasks + tools + tests).
118+
All MetaSim-side changes are opt-in (`SimParamCfg` knobs default to `None`/`False`; the new
119+
`PrimitiveMultiBoxCfg`, `get_pairwise_contact_force`, and `PrimitiveCubeCfg`
120+
`fix_base_link` handling are additive) — existing SAPIEN3 tasks are byte-identical
121+
(417 sapien3 + general MetaSim tests pass). The RoboVerse side is purely additive
122+
(`_native/` package + tasks + tools + tests). Both MetaSim and RoboVerse changes are merged
123+
to their respective public `main` branches.
102124

103125
## Remaining
104126

105-
`peg_insertion_side` / `plug_charger` success use ManiSkill internal pose
106-
helpers (peg-in-hole / charger-to-goal) and are not yet ported (geometric
107-
proxy in place); their dense rewards (and demo-`.h5` replay, non-gripper
108-
robots like PushT/DrawTriangle, and multi-agent TwoRobot tasks) are
109-
tracked as follow-ups.
127+
`peg_insertion_side` and `plug_charger` randomize their **internal geometry** per episode
128+
(the box-hole position / charger prong layout). Their success/reward formulas are exact
129+
and their part *poses* randomize per episode, but reproducing the per-episode internal
130+
geometry would require rebuilding the multi-box collision actor each reset (framework-
131+
atypical); it's left as a follow-up to preserve backward compatibility.

0 commit comments

Comments
 (0)