Skip to content

[BugFix] Stabilize LIBERO VLA GRPO training#3940

Merged
vmoens merged 10 commits into
gh/vmoens/306/basefrom
gh/vmoens/306/head
Jul 6, 2026
Merged

[BugFix] Stabilize LIBERO VLA GRPO training#3940
vmoens merged 10 commits into
gh/vmoens/306/basefrom
gh/vmoens/306/head

Conversation

@vmoens

@vmoens vmoens commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Stack from ghstack (oldest at bottom):

Add reference rollout comparison and parity diagnostics for OpenVLA on LIBERO.

Align preprocessing, CPU action decoding, inference microbatching, collector lifecycle, replay grouping, evaluator metrics, and policy synchronization with the validated runtime.

Add paper-shaped configuration controls and focused regression coverage for the end-to-end VLA GRPO path.

@vmoens
Update
5679bf1 
[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3940

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 6, 2026
Open
Merged
Merged
Merged
Merged
Merged
Merged
Closed
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 6, 2026
@vmoens
Update
3e73b46 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[BugFix] Stabilize LIBERO VLA GRPO training
1e384fd 
Add reference rollout comparison and parity diagnostics for OpenVLA on LIBERO.

Align preprocessing, CPU action decoding, inference microbatching, collector lifecycle, replay grouping, evaluator metrics, and policy synchronization with the validated runtime.

Add paper-shaped configuration controls and focused regression coverage for the end-to-end VLA GRPO path.


ghstack-source-id: a88d47e
Pull-Request: #3940
@github-actions

github-actions Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Results: PR 90fa0408 vs main 25f04547

Benchmark run: https://github.com/pytorch/rl/actions/runs/28797325339

Higher ops/sec is better. Tables are sorted by largest absolute change.

CPU

Compared 216 benchmarks. Regressions over 5%: 11. Improvements over 5%: 14.

Benchmark main ops PR ops Change
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] 56.21 84.39 +50.13%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 56.87 83.16 +46.24%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] 61.90 88.99 +43.77%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td_lambda_return_estimate-True-False] 56.43 77.36 +37.08%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 350.15 415.77 +18.74%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 922.99 1,069 +15.82%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 29.32 24.86 -15.21%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 20.25 23.21 +14.62%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 110.65 126.23 +14.08%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 741.31 832.77 +12.34%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 701.17 776.23 +10.71%
benchmarks/test_collectors_benchmark.py::test_single_with_rb 7.3482 6.5851 -10.38%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,599 3,245 -9.82%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,603 3,267 -9.33%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] 6.4791 7.0673 +9.08%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 36.11 39.38 +9.06%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 99.94 91.15 -8.80%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,520 3,210 -8.79%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-False] 44,029 47,716 +8.37%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,340 3,095 -7.33%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,219 2,069 -6.77%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 265.22 282.21 +6.41%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,212 3,035 -5.50%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,273 3,103 -5.20%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,104 1,996 -5.16%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False] 63,848 67,035 +4.99%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,758 1,846 +4.96%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 268.70 281.93 +4.93%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,269 3,430 +4.93%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-backward] 65.69 62.60 -4.69%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] 23,253 24,302 +4.51%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 233.60 244.01 +4.45%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,015 2,104 +4.42%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 954.20 995.88 +4.37%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-backward] 26.86 28.02 +4.32%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] 439.47 458.30 +4.28%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] 2.9324 3.0540 +4.14%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 857.22 892.47 +4.11%
benchmarks/test_envs_benchmark.py::test_simple 1.7849 1.7133 -4.01%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cpu] 97.02 93.24 -3.90%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 637.96 662.12 +3.79%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 34,100 35,363 +3.70%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] 6,976 7,231 +3.65%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 577.29 556.29 -3.64%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] 4,260 4,412 +3.56%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 22.95 23.76 +3.55%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] 18,736 19,398 +3.54%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 109.87 113.63 +3.42%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] 14.68 15.17 +3.33%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 79.66 82.27 +3.28%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 41.97 43.32 +3.22%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 19,833 20,470 +3.21%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[50-img_shape0-small] 3,454 3,564 +3.20%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 49.94 51.50 +3.13%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,930 3,022 +3.13%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] 4,127 4,001 -3.05%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-True-0-gru] 1.4187 1.4617 +3.03%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 531.56 516.70 -2.79%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] 27,023 27,757 +2.72%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] 33,826 34,722 +2.65%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,007 1,955 -2.62%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False] 28,891 29,629 +2.55%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 114.05 116.96 +2.55%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 7,948 7,748 -2.52%
benchmarks/test_envs_benchmark.py::test_serial 0.5767 0.5911 +2.49%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 28,326 29,031 +2.49%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 21,689 22,224 +2.46%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] 17,885 18,325 +2.46%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 192.18 196.90 +2.45%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-backward] 53.92 55.23 +2.43%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] 38,637 37,700 -2.42%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 332.43 340.10 +2.31%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 24.85 25.42 +2.28%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] 252.69 258.43 +2.27%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[reduce-overhead-None] 222.61 217.59 -2.25%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 20,037 19,606 -2.15%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 513.76 524.60 +2.11%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 274.09 279.72 +2.05%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] 31,552 32,192 +2.03%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 24.13 24.62 +2.03%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 0.5912 0.6032 +2.02%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 664.45 651.28 -1.98%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] 351,165 357,949 +1.93%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-True-0-lstm] 0.9422 0.9599 +1.88%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 114.02 116.16 +1.88%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] 161.70 164.67 +1.83%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 133.81 136.26 +1.83%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 309.96 315.42 +1.76%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 38,020 38,688 +1.76%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 509.77 518.72 +1.76%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-None] 37.04 37.68 +1.73%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] 695.02 706.97 +1.72%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-None] 93.31 94.91 +1.71%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] 4.4186 4.4931 +1.69%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] 87.64 89.11 +1.68%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True] 1.3193 1.2971 -1.68%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 5,124 5,206 +1.60%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[100-img_shape0-atari] 20.09 20.40 +1.58%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 565.80 557.03 -1.55%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] 338.91 333.69 -1.54%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] 49,911 50,661 +1.50%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 56,844 57,696 +1.50%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] 121.24 123.03 +1.48%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 37,857 38,406 +1.45%
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] 25.31 24.94 -1.45%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] 35.87 36.38 +1.43%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] 82.15 83.31 +1.41%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 274.84 278.60 +1.37%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru] 1.3875 1.4062 +1.35%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.5201 0.5268 +1.30%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] 89.73 90.90 +1.29%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] 30,479 30,871 +1.29%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] 196.66 194.17 -1.27%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 20,821 21,076 +1.23%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 163.51 161.54 -1.21%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] 208.45 210.93 +1.19%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None] 120.85 122.28 +1.18%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] 31,668 32,042 +1.18%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 168.20 166.23 -1.18%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 42,211 42,703 +1.16%
... ... ... Showing 120 of 216 comparisons, sorted by absolute change.

GPU

Compared 226 benchmarks. Regressions over 5%: 10. Improvements over 5%: 8.

Benchmark main ops PR ops Change
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 60.17 87.71 +45.78%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 76.78 102.85 +33.95%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 84.75 105.09 +24.00%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,936 2,351 -19.94%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 44.70 37.71 -15.64%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,816 2,448 -13.09%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,037 3,420 +12.61%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 534.16 467.60 -12.46%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 817.52 731.26 -10.55%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 701.49 770.11 +9.78%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 3,936 4,247 +7.89%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 946.87 1,015 +7.18%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,798 2,611 -6.70%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] 756.12 708.41 -6.31%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 507.75 477.46 -5.97%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,914 2,749 -5.67%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,292 2,165 -5.55%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 41.14 43.30 +5.24%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 545.72 518.62 -4.97%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] 4,369 4,161 -4.76%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 38,158 36,354 -4.73%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] 150.44 143.51 -4.61%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 275.99 263.88 -4.39%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] 4,665 4,461 -4.37%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 711.00 680.26 -4.32%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None] 344.29 329.73 -4.23%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] 76.38 73.16 -4.23%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 984.23 1,024 +4.07%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 406.55 390.15 -4.03%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,280 2,189 -4.00%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 390.06 374.58 -3.97%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,529 3,664 +3.82%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] 32,682 31,443 -3.79%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] 8.5957 8.9201 +3.77%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] 34,532 33,242 -3.74%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 745.35 772.44 +3.63%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.4991 0.4812 -3.59%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-None] 53.92 52.09 -3.40%
benchmarks/test_envs_benchmark.py::test_parallel 0.5335 0.5512 +3.33%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 495.79 479.47 -3.29%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] 82.17 79.52 -3.23%
benchmarks/test_collectors_benchmark.py::test_sync_pixels 10.26 9.9325 -3.22%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 5.4841 5.3111 -3.15%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64] 12.09 12.46 +3.09%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 38,476 37,293 -3.08%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] 37,280 36,155 -3.02%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] 49,813 48,325 -2.99%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 393.06 404.58 +2.93%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 889.62 863.57 -2.93%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 42,532 41,296 -2.91%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 651.63 633.19 -2.83%
benchmarks/test_envs_benchmark.py::test_serial 0.4176 0.4293 +2.80%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 22.68 23.31 +2.76%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] 10.51 10.79 +2.75%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-1] 496.94 510.12 +2.65%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-16] 47.70 48.96 +2.65%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] 15.05 14.66 -2.61%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 51.63 52.97 +2.59%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 22,110 21,538 -2.58%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 729.02 747.58 +2.55%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 1,300 1,333 +2.53%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward] 232.54 226.67 -2.53%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False] 64,085 62,482 -2.50%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 562.10 548.10 -2.49%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 57,453 56,027 -2.48%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[50-img_shape0-small] 3,546 3,458 -2.47%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] 35.49 36.35 +2.42%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] 19,267 19,709 +2.30%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] 22,880 23,384 +2.20%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] 283.42 277.20 -2.19%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] 598.62 611.45 +2.14%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[100-img_shape0-atari] 20.45 20.01 -2.14%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,128 2,172 +2.10%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,876 1,838 -2.04%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 267.20 272.61 +2.03%
benchmarks/test_envs_benchmark.py::test_simple 1.2189 1.1946 -1.99%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] 31,504 30,884 -1.97%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False] 28,963 28,427 -1.85%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 350.14 356.47 +1.81%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 47.55 48.40 +1.80%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 20,470 20,102 -1.80%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,063 3,011 -1.70%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 172.83 175.72 +1.67%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 51.92 52.79 +1.66%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 19,538 19,215 -1.65%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] 191.67 194.83 +1.65%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] 467.31 459.61 -1.65%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] 681.01 692.19 +1.64%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] 358,650 364,497 +1.63%
benchmarks/test_envs_benchmark.py::test_transformed 0.7064 0.6950 -1.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 161.94 164.52 +1.60%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari] 29.91 29.43 -1.59%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] 47.97 47.20 -1.59%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None] 113.08 111.28 -1.59%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 77,181 75,970 -1.57%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb_cuda[100-img_shape0-atari] 13.28 13.07 -1.57%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] 34,438 33,899 -1.57%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 11,451 11,629 +1.56%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] 71.63 70.52 -1.55%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] 47.87 48.61 +1.54%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] 76.66 75.49 -1.53%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] 277.44 273.21 -1.53%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 0.5899 0.5989 +1.52%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] 6,055 5,965 -1.49%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] 745.66 734.53 -1.49%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 331.23 326.34 -1.48%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] 31,761 31,295 -1.47%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[200-img_shape1-large_batch] 8.6319 8.5078 -1.44%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] 118.96 120.66 +1.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 193.39 196.15 +1.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,208 2,239 +1.42%
benchmarks/test_collectors_benchmark.py::test_single_pixels 6.2706 6.1821 -1.41%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 1,344 1,363 +1.40%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-triton-False-0-gru] 20.67 20.38 -1.40%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 10.44 10.58 +1.33%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False] 63,258 62,427 -1.31%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] 803.98 814.51 +1.31%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[200-img_shape1-large_batch] 10.25 10.11 -1.31%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] 158.15 156.09 -1.30%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] 229.78 226.80 -1.30%
... ... ... Showing 120 of 226 comparisons, sorted by absolute change.

@vmoens
Update
d2e496b 
[ghstack-poisoned]
This was referenced Jul 6, 2026
Merged
Merged
vmoens added 4 commits July 6, 2026 12:56
@vmoens
Update
e3a7440 
[ghstack-poisoned]
@vmoens
Update
2e5f1d5 
[ghstack-poisoned]
@vmoens
Update
e785390 
[ghstack-poisoned]
@vmoens
Update
d4d19cc 
[ghstack-poisoned]
vmoens added 2 commits July 6, 2026 13:37
@vmoens
Update
0124692 
[ghstack-poisoned]
@vmoens
Update
e47e412 
[ghstack-poisoned]
@vmoens
Update
90fa040 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[BugFix] Stabilize LIBERO VLA GRPO training
25f9361 
Align preprocessing, CPU action decoding, inference microbatching, collector
lifecycle, replay grouping, evaluator metrics, and policy synchronization with
the validated runtime.

Add paper-shaped configuration controls and focused regression coverage for
the end-to-end VLA GRPO path. Absorbs the async collection-mode work
(per-worker group-id offsets, mode-selection plumbing) whose earlier
prototype implementation this commit replaces.

Review fixes: version stamping and weight updates run under the model lock;
the control-plane timeout raises TimeoutError; the tokenizer decode path
works on CUDA-resident modules, returns float32 on the input device, and
drops its unreachable tensor branch; the evaluator warns on missing
trajectory info instead of silently reporting one episode; async eval no
longer stalls on shutdown when nothing is pending; thread-backend video
recording refuses to silently narrow multi-task evals; the sbatch launch
matches its resource request; weight sync ships trainable-only parameters by
default; dead config keys removed and README metrics/trajectory counts
corrected.

ghstack-source-id: 8e94422
Pull-Request: #3940
# Conflicts:
#	torchrl/modules/inference_server/_server.py
@vmoens vmoens merged commit 90fa040 into gh/vmoens/306/base Jul 6, 2026
99 of 104 checks passed
@vmoens vmoens deleted the gh/vmoens/306/head branch July 6, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

BugFix CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Collectors Integrations/torch_geometric Integrations llm/ LLM-related PR, triggers LLM CI tests Modules Objectives ReplayBuffers sota-implementations/ Transforms

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vmoens