On Inconsistent Training Results of Identical Training Tasks

[alexsneering]

While training the Unitree-H1-Velocity task from unitree_rl_lab in Isaac Lab, I did not modify any hyperparameters at all. I ran two separate training runs on the identical task setup, yet the corresponding TensorBoard plots fail to overlap. What could be the root cause? From my prior experience working with Isaac Gym, training the same task with identical fixed random seeds should yield perfectly matching training curves.

[yudin-s]

I would not expect TensorBoard curves from two PPO training runs to overlap perfectly just because the task config and seed match. Isaac Lab can seed the environment and agent config, but GPU physics, PyTorch/CUDA kernels, parallel resets, and optimizer updates can still introduce tiny differences that compound quickly.

A useful separation is: deterministic environment transitions are a narrower claim than deterministic end-to-end RL training. Isaac Lab has environment determinism tests for fixed observations/rewards, but the RSL-RL training script also sets the env seed from the agent seed, changes it per rank for distributed runs, and trains with randomized initial episode lengths.

I would check in this order:

1. Run both jobs with the exact same --seed, --num_envs, --device, Isaac Sim/Isaac Lab versions, and no --distributed.
2. Compare the saved params/env.yaml and params/agent.yaml from both log directories.
3. Compare the first few rollout observations/rewards, or a short fixed-action rollout. If those differ immediately, look at task randomization, seed propagation, and device setup.
4. If early rollouts match but TensorBoard curves drift later, treat it as normal RL sensitivity rather than a configuration mismatch.

If you need bitwise repeatability for an experiment, also avoid changing GPU model/driver and check whether the backend you use exposes deterministic Torch settings. Even then, Isaac Gym-style perfectly overlapping curves are not something I would assume for the whole Isaac Lab training stack.

[alexsneering]

Okay. Following your instructions, I specified --seed, --num_envs and --device, then ran training twice on Unitree-G1-29dof-Velocity. I found the TensorBoard curves overlap perfectly. After inspection, both runs used num_envs = 4096, so the discrepancy likely stems from differing seeds or device configurations. Thanks for your help!

[yudin-s]

Great, that result is useful. It means the first thing to trust is the saved run configuration: once --seed, --num_envs, and --device were pinned consistently, this task did behave deterministically enough for overlapping curves.

I would keep those three values explicit in future comparison runs and compare the saved params files whenever curves diverge. If they differ there, it is a run-setup mismatch; if the saved params match and curves still diverge, then it is worth looking deeper at backend nondeterminism.