Skip to content

[Feature] Track behavior policy versions in inference server#3895

Merged
vmoens merged 13 commits into
gh/vmoens/290/basefrom
gh/vmoens/290/head
Jul 6, 2026
Merged

[Feature] Track behavior policy versions in inference server#3895
vmoens merged 13 commits into
gh/vmoens/290/basefrom
gh/vmoens/290/head

Conversation

@vmoens

@vmoens vmoens commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@vmoens
Update
985c7d0 
[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3895

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions github-actions Bot added Feature New feature Documentation Improvements or additions to documentation Collectors Modules Integrations/torch_geometric Integrations and removed Feature New feature labels Jun 21, 2026
This was referenced Jun 21, 2026
Merged
Merged
Merged
Merged
@github-actions github-actions Bot added the Feature New feature label Jun 21, 2026
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Results: PR c4999361 vs main 925332ee

Benchmark run: https://github.com/pytorch/rl/actions/runs/28797325335

Higher ops/sec is better. Tables are sorted by largest absolute change.

CPU

Compared 216 benchmarks. Regressions over 5%: 11. Improvements over 5%: 33.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 50.25 172.26 +242.79%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 199.64 38.50 -80.71%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] 60.89 90.52 +48.66%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,775 3,459 +24.64%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] 36.19 28.73 -20.63%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1,971 2,314 +17.40%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 491.20 564.50 +14.92%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 776.11 890.72 +14.77%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 250.52 287.50 +14.76%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 29.70 25.35 -14.63%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 112.45 128.32 +14.11%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,052 3,473 +13.80%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 379.85 428.61 +12.83%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1,859 2,087 +12.32%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,288 3,680 +11.92%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,156 3,532 +11.91%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 543.29 482.22 -11.24%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 32.82 29.14 -11.21%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[True-None] 205.90 228.09 +10.78%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] 427.01 468.48 +9.71%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] 7.1648 6.4977 -9.31%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 766.63 837.82 +9.29%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] 739.77 801.65 +8.37%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 232.55 251.50 +8.15%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 129.29 139.82 +8.14%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 78.44 84.77 +8.07%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,189 2,948 -7.56%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 378.56 405.74 +7.18%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] 10.76 9.9974 -7.05%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 727.58 778.66 +7.02%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] 54.99 58.80 +6.93%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 110.98 118.60 +6.87%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] 113.12 120.08 +6.16%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 109.45 115.84 +5.84%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] 7,068 7,472 +5.72%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] 451.35 476.99 +5.68%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 399.70 421.48 +5.45%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 672.58 708.88 +5.40%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] 539.64 568.50 +5.35%
benchmarks/test_envs_benchmark.py::test_simple 1.8431 1.7452 -5.31%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] 51,114 53,760 +5.18%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] 57.68 60.66 +5.17%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] 12,719 12,069 -5.12%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1,110 1,053 -5.11%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 16.85 16.02 -4.90%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[reduce-overhead-None] 559.14 585.83 +4.77%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] 688.50 721.31 +4.77%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1,073 1,124 +4.71%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 550.02 574.93 +4.53%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 2,236 2,337 +4.52%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[reduce-overhead-None] 702.75 734.44 +4.51%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 58,976 56,375 -4.41%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru] 1.3962 1.3350 -4.38%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 22,140 21,226 -4.13%
benchmarks/test_envs_benchmark.py::test_transformed 0.9274 0.8892 -4.12%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 516.06 496.13 -3.86%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 37,401 38,783 +3.69%
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] 26.46 25.51 -3.59%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,796 1,860 +3.54%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] 281.17 271.28 -3.52%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 167.11 172.81 +3.41%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 272.59 281.66 +3.33%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] 34,203 33,115 -3.18%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,254 3,151 -3.18%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] 82.25 84.76 +3.05%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] 9.1396 8.8644 -3.01%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] 57,500 59,207 +2.97%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 324.64 334.21 +2.95%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] 468.24 481.85 +2.91%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] 1.9908 2.0485 +2.90%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] 123.07 126.57 +2.84%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-backward] 32.51 33.43 +2.83%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] 261.01 267.88 +2.63%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cpu] 51.88 53.24 +2.62%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] 4.4444 4.5606 +2.62%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 305.94 313.85 +2.59%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] 332.91 341.50 +2.58%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 7,652 7,847 +2.54%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 280.30 287.34 +2.51%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-lstm] 0.8695 0.8477 -2.50%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[reduce-overhead-None] 225.13 230.65 +2.45%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 555.58 569.11 +2.44%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-None] 270.34 276.91 +2.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 26.07 25.43 -2.42%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-None] 49.33 50.52 +2.40%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] 1,774 1,732 -2.35%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] 176.86 180.93 +2.30%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] 160.00 163.68 +2.30%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 139.44 136.26 -2.28%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 633.28 647.49 +2.24%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 169.86 173.64 +2.22%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] 32,349 33,067 +2.22%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-backward] 78.12 79.84 +2.21%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] 4.2033 4.2951 +2.18%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 519.16 530.34 +2.15%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] 28,067 28,666 +2.14%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 20,495 20,070 -2.07%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] 214.85 219.24 +2.04%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] 17.85 18.21 +2.02%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,930 2,872 -2.00%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 32,660 32,010 -1.99%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 0.6092 0.6213 +1.99%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 4,853 4,757 -1.97%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 201.85 198.00 -1.91%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 164.40 167.48 +1.88%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] 54.91 55.93 +1.87%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,139 2,100 -1.85%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 21,508 21,117 -1.82%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-backward] 62.72 63.82 +1.76%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] 18,599 18,925 +1.75%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] 19.74 20.08 +1.73%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td_lambda_return_estimate-True-False] 55.09 56.01 +1.66%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] 83.51 84.90 +1.66%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] 19,258 19,571 +1.63%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 115.97 117.83 +1.60%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,212 3,161 -1.59%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] 133.06 135.16 +1.57%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] 92.08 93.52 +1.56%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 266.12 270.25 +1.55%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 23.67 23.31 -1.51%
... ... ... Showing 120 of 216 comparisons, sorted by absolute change.

GPU

Compared 226 benchmarks. Regressions over 5%: 10. Improvements over 5%: 19.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 43.70 197.05 +350.91%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 185.18 49.08 -73.50%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 60.53 87.79 +45.04%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 76.56 101.71 +32.84%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 85.14 105.25 +23.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,427 2,633 -23.17%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1,898 2,334 +22.95%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 712.19 843.49 +18.44%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,055 3,546 +16.08%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 437.71 500.55 +14.36%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,913 3,299 +13.27%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 989.26 860.44 -13.02%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1,993 2,230 +11.88%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 977.15 870.29 -10.94%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 788.94 704.09 -10.75%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 497.31 539.16 +8.41%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 956.68 1,024 +7.00%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] 3,966 4,240 +6.91%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] 24,589 22,908 -6.83%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 347.18 370.52 +6.72%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,215 2,349 +6.02%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,002 1,889 -5.65%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 525.57 496.10 -5.61%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 277.47 262.07 -5.55%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] 48.74 46.05 -5.51%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 40.79 43.02 +5.46%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,337 3,517 +5.41%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] 19,070 20,080 +5.30%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 802.89 844.35 +5.16%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 322.46 338.51 +4.98%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] 47,330 49,587 +4.77%
benchmarks/test_envs_benchmark.py::test_simple 1.2483 1.1910 -4.59%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 28,231 29,484 +4.44%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 707.70 677.23 -4.31%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] 75.29 78.36 +4.07%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] 621.48 646.22 +3.98%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] 48,263 50,137 +3.88%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] 22,880 23,752 +3.81%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 33,888 35,154 +3.73%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] 743.23 715.94 -3.67%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] 48.76 50.55 +3.66%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 134.32 139.19 +3.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,869 2,973 +3.61%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 9.9967 10.36 +3.59%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] 18,003 18,647 +3.58%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 468.67 451.93 -3.57%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 4,429 4,276 -3.45%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 20,383 21,081 +3.43%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] 27,973 27,019 -3.41%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False] 62,180 64,280 +3.38%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari] 30.37 29.36 -3.34%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-memmap_cpu_storage_cud... 964.40 996.06 +3.28%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 36,554 37,737 +3.23%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] 634.86 614.53 -3.20%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 74,816 77,080 +3.03%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 733.84 755.32 +2.93%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] 54,057 55,533 +2.73%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 37,341 38,320 +2.62%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] 6,024 6,179 +2.58%
benchmarks/test_envs_benchmark.py::test_transformed 0.7071 0.6894 -2.51%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 0.5912 0.6057 +2.44%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] 19.75 20.21 +2.36%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] 15.31 14.94 -2.36%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None] 781.68 763.27 -2.36%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,721 2,784 +2.31%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward] 240.63 235.10 -2.30%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.5097 0.4980 -2.30%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] 853.95 873.36 +2.27%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] 684.14 699.42 +2.23%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] 20.76 21.22 +2.22%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None] 339.08 331.63 -2.20%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 29,610 30,259 +2.19%
benchmarks/test_collectors_benchmark.py::test_async 11.05 10.82 -2.14%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 55,708 56,851 +2.05%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-16] 48.08 49.02 +1.96%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64] 1.2261 1.2500 +1.95%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-False] 0.2205 0.2247 +1.91%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 23.72 24.17 +1.91%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] 17,580 17,914 +1.90%
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] 19.82 20.19 +1.87%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] 34,338 34,974 +1.85%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cuda] 2,201 2,241 +1.83%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-serial-buffers-True] 0.5103 0.5195 +1.81%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] 279.82 284.86 +1.80%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True] 1.2999 1.3233 +1.79%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] 8.7176 8.8730 +1.78%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,609 3,673 +1.76%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 468.52 476.68 +1.74%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] 155.35 158.05 +1.74%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] 371,930 378,386 +1.74%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 41,718 42,437 +1.72%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] 98.90 100.59 +1.71%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 48.64 47.82 -1.70%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 165.97 168.79 +1.70%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari] 17.16 17.46 +1.70%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] 10.46 10.64 +1.67%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 19,348 19,668 +1.65%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 47.32 48.09 +1.64%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16] 11.93 12.13 +1.62%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp... 1,473 1,497 +1.62%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 554.99 546.27 -1.57%
benchmarks/test_collectors_benchmark.py::test_async_pixels 10.84 10.67 -1.57%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] 35.69 36.24 +1.56%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 163.85 166.36 +1.53%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] 33,749 34,267 +1.53%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[reduce-overhead-None] 44.89 44.20 -1.53%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] 41,697 42,331 +1.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] 31,475 31,952 +1.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 19,909 20,207 +1.50%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 159.77 162.15 +1.49%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-4] 47.92 48.63 +1.47%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-4] 181.68 184.24 +1.41%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 274.10 277.96 +1.41%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 307.92 303.61 -1.40%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 21.55 21.85 +1.37%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-False] 1.5840 1.6049 +1.32%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] 231.88 234.86 +1.29%
benchmarks/test_collectors_benchmark.py::test_single_with_rb_pixels 5.3511 5.2836 -1.26%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] 393.58 398.49 +1.25%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-1] 513.24 506.83 -1.25%
... ... ... Showing 120 of 226 comparisons, sorted by absolute change.

vmoens added 3 commits June 22, 2026 09:18
@vmoens
Update
e374b9b 
[ghstack-poisoned]
@vmoens
Update
8eb060c 
[ghstack-poisoned]
@vmoens
Update
1cd821c 
[ghstack-poisoned]
This was referenced Jun 24, 2026
Merged
Closed
theap06
theap06 suggested changes Jun 27, 2026
View reviewed changes
Comment thread torchrl/modules/inference_server/_server.py
Comment thread torchrl/collectors/_async_batched.py Outdated
@vmoens
Update
febe53f 
[ghstack-poisoned]
This was referenced Jul 6, 2026
Open
Merged
Merged
@vmoens vmoens marked this pull request as ready for review July 6, 2026 08:59
vmoens added 2 commits July 6, 2026 10:04
@vmoens
Update
cd9fd3b 
[ghstack-poisoned]
@vmoens
Update
604e7b5 
[ghstack-poisoned]
@vmoens vmoens mentioned this pull request Jul 6, 2026
Merged
@vmoens vmoens mentioned this pull request Jul 6, 2026
Merged
vmoens added 2 commits July 6, 2026 12:56
@vmoens
Update
66056ae 
[ghstack-poisoned]
@vmoens
Update
8131680 
[ghstack-poisoned]
vmoens added 4 commits July 6, 2026 13:21
@vmoens
Update
c6323d2 
[ghstack-poisoned]
@vmoens
Update
09ea51b 
[ghstack-poisoned]
@vmoens
Update
9cf2250 
[ghstack-poisoned]
@vmoens
Update
c499936 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Track behavior policy versions in inference server
cd1406e 
ghstack-source-id: 048a08f
Pull-Request: #3895
# Conflicts:
#	test/test_inference_server.py
#	torchrl/modules/inference_server/_server.py
# Conflicts:
#	torchrl/collectors/_async_batched.py
#	torchrl/modules/inference_server/_client.py
@vmoens vmoens merged commit c499936 into gh/vmoens/290/base Jul 6, 2026
101 of 104 checks passed
@vmoens vmoens deleted the gh/vmoens/290/head branch July 6, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@theap06 theap06 theap06 requested changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Collectors Documentation Improvements or additions to documentation Feature New feature Integrations/torch_geometric Integrations Modules ReplayBuffers Transforms

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vmoens @theap06