Robot Hand/Fingers Penetrating Cube and Unable to Grasp or Lift Object in Isaaclab

[Ihebzaouali]

Question

Hello,
I am working on a manipulation task in Isaaclab where a custom humanoid robot needs to pick and place a cube into a bin.

My current problem is that the fingers/hand penetrate the cube and sometimes even parts of the scene (cube/bin). The robot closes its fingers, but it cannot properly grasp or lift the cube only when being penetrated in the cube.

Things I already tried:
-Adjusting PhysX solver position iteration count
-Adjusting solver velocity iteration count
-Adding and tuning static/dynamic friction on both the fingers and the cube
-Checking/adding contact and collision settings on fingers and cube
-Adding a Surface Gripper to the end effector

Despite these changes, the issue still happens:
-Fingers penetrate the cube
-Surface gripper not working

Build Info

• Isaac Lab Version: 2.3.2
• Isaac Sim Version: 5.1

[RandomOakForest]

Thank you for posting this. You are hitting a combination of contact settings, geometry, and control issues. The good news is that Isaac Lab/Isaac Sim can do robust grasps; you need a few specific tweaks beyond solver iterations and friction.¹²³

Below is a focused checklist I’d go through; you can try them in order.

---

1. Fix basic collision/contact parameters

On both the cube and each finger collider, explicitly set:

• Contact offset
  • Many default imports end up with offsets that are too small, which lets bodies interpenetrate before contacts are resolved.¹
  • Try:
    • contactOffset in the range $0.005–0.01$ m (5–10 mm).¹
• Rest offset
  • If your collision shapes are slightly bigger than the visual mesh (typical), use a negative rest offset so contact stabilizes near the visual surface.¹
  • Example: restOffset = -0.001 to -0.002 m.

Where to set this in Isaac Lab:

• In your asset USD: the collider prims for finger links and cube links (e.g., Physics → Colliders → contactOffset, restOffset).¹
• Or via Isaac Lab config if you generate assets procedurally, using the collider/rigid-body utility configs with those fields overridden.⁴

Also check:

• Scale: confirm your cube is not tiny (e.g., 1 cm) relative to the robot; small objects are more prone to tunneling.³
• Collision type: prefer primitive or convex colliders (box for cube, simple convex approximations for fingers), not high‑poly triangle meshes.²

---

2. Improve simulation step and solver precision

Beyond solver iterations, step timing matters a lot for grasping.²¹

For your SimulationCfg (or equivalent Isaac Lab sim config), try:

• dt = 0.005 (from 0.01)
• substeps = 4
• solver_iterations = 64 (or keep your high iteration count but pair it with smaller dt).¹

This effectively gives more, smaller physics steps per control step and greatly reduces “tunneling” where fingers pass through the cube between frames.²¹

If your control loop runs slower than the physics loop (e.g., control at 20 Hz, physics at 200 Hz), that’s fine; just make sure joint targets aren’t changing too aggressively frame to frame.

---

3. Change how you drive the hand joints

If you close the fingers with very stiff position control and large position steps, they can “punch” through the cube before forces resolve.²¹

Try instead:

• Use velocity control with force limits for finger closure, at least during grasp.¹
  • Small closing velocities (e.g., a few rad/s) and reasonable torque/force limits.
• If you must use position control:
  • Move in small increments toward closed configuration.
  • Set joint stiffness and damping so the effective contact forces are not ridiculously high (no “infinite stiffness” behavior).²

Conceptually: you want fingers to “push until resistance” and stop, not to teleport through the cube.

---

4. Verify collision layers and self-collision

Penetration into the bin or scene may indicate some prims are missing colliders or on non‑interacting collision groups.²

Check:

• Cube:
  • Has RigidBody enabled and a proper collider.⁵
  • Collision group/mask actually interacts with both finger colliders and bin colliders.
• Bin:
  • Has colliders on walls and bottom; no gaps.²
• Robot:
  • Self‑collision generally enabled for fingers/palm, except where you intentionally disable it.

In Isaac Sim/Isaac Lab GUI:

• Enable Physics → Show → Colliders to inspect shape extents and make sure fingers are not already starting inside the cube or bin collider at the beginning of grasp.⁵

---

5. Making the Surface Gripper actually work

Given you already tried a Surface Gripper, I’d treat it as a “snap” attachment after contact, not a replacement for finger collision. It often fails because of small configuration details.⁵²

Minimum checks (based on 5.x behavior, still relevant conceptually):

1. Schema and joints
   • The gripper prim (usually at the fingertip or palm) has the SurfaceGripper schema applied.⁵
   • Attachment Points lists your D6 attachment joints with absolute prim paths.⁵
   • Each attachment joint is D6, enabled, and located at the contact region, not inside the finger or far away.⁵
2. Raycast and clearance
   • The Forward Axis points out of the fingertip toward the cube, not into the hand.⁵
   • Clearance Offset is big enough so that the raycast origin begins outside any fingertip collider; otherwise the gripper “hits itself” and never sees the cube.⁵
   • Check this with Physics → Show → Colliders and the joint gizmos visible.⁵
3. Target object is grippable
   • The cube is a separate rigid body (not part of the robot articulation).⁵
   • It has a collider and lies within Max Grip Distance when you close.⁵
4. Control signals / update
   • Ensure you are actually sending enable=True and the close command for multiple simulation steps, and that your surface gripper update function is called every frame, consistent with official examples.⁵
   • Watch the Surface Gripper status fields: if it never reaches “Closed” or “Grasped”, the raycast is not finding the cube.⁵

As a debugging step: take the working Surface Gripper example from Isaac Sim/Isaac Lab, mount that exact gripper (with its schema and joints) onto your humanoid’s end effector, and see if it behaves correctly there.³⁵
If it works, the problem is your custom gripper config; if not, the issue is more global (physics step, layers, etc.).

---

6. Cross-check with a known-good pick-and-place

Isaac Lab includes pick‑and‑place environments with humanoids picking cubes into baskets/bins (e.g., Unitree G1 demo).⁴³

I’d do:

• Run a stock pick-and-place environment from the Isaac Lab showroom that uses object pickup and contact.⁴³
• Compare:
  • Object/collider sizes
  • contactOffset / restOffset values
  • Simulation dt, substeps, solver iterations
  • Hand joint drive modes and limits

This gives you a concrete reference for parameters that actually work in a similar scenario.³⁴

---

7. If nothing works: temporary “fake grasp”

For debugging training pipelines, some teams temporarily bypass physics to confirm the rest of the task works:²

• On first successful contact / close command, programmatically create a fixed joint between the cube and the end-effector frame, or directly parent the cube to the hand in the scene graph and disable physics on it.²
• Later, switch back to full physical grasp once contact and control are tuned.

This is not physically exact, but it helps you isolate whether your RL / control stack is correct versus physics settings.

Footnotes

1. https://github.com/isaac-sim/IsaacLab/discussions/2765 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰

2. https://forums.developer.nvidia.com/t/how-to-improve-stability-when-grasping-an-object/221959 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹

3. https://isaac-sim.github.io/IsaacLab/main/source/overview/showroom.html ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

4. https://isaac-sim.github.io/IsaacLab/main/source/overview/environments.html ↩ ↩² ↩³ ↩⁴

5. https://forums.developer.nvidia.com/t/surface-gripper-in-5-1-not-working-using-gui/355568 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴