Skip to content

[Feature] Add remote policy module for inference server clients#3894

Merged
vmoens merged 10 commits into
gh/vmoens/289/basefrom
gh/vmoens/289/head
Jul 6, 2026
Merged

[Feature] Add remote policy module for inference server clients#3894
vmoens merged 10 commits into
gh/vmoens/289/basefrom
gh/vmoens/289/head

Conversation

@vmoens

@vmoens vmoens commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@vmoens
Update
c633f45 
[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3894

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jun 21, 2026
Merged
Merged
Merged
Merged
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Results: PR 40b95d0c vs main 23c74287

Benchmark run: https://github.com/pytorch/rl/actions/runs/28797047196

Higher ops/sec is better. Tables are sorted by largest absolute change.

CPU

Compared 216 benchmarks. Regressions over 5%: 19. Improvements over 5%: 19.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 49.28 190.65 +286.87%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 194.38 38.20 -80.35%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] 61.21 90.17 +47.31%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td_lambda_return_estimate-True-False] 74.37 55.29 -25.66%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,797 2,867 -24.50%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,785 2,875 -24.04%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,514 2,687 -23.53%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,570 2,811 -21.27%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,517 2,780 -20.94%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] 68.95 55.15 -20.02%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 217.83 257.41 +18.17%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 67.43 55.22 -18.12%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,290 1,891 -17.39%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,394 1,990 -16.89%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 29.42 24.73 -15.95%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,042 2,359 +15.53%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 34.02 28.86 -15.17%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 20.36 23.18 +13.82%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 550.27 476.45 -13.42%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 883.48 995.54 +12.68%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 865.97 758.70 -12.39%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 810.09 720.00 -11.12%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] 111.48 123.74 +10.99%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 93.07 102.80 +10.45%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] 647.30 711.95 +9.99%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] 56.34 61.90 +9.87%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-1] 79.04 71.38 -9.69%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] 9.7200 10.63 +9.38%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1,023 1,115 +8.92%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] 79.23 85.97 +8.50%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] 631.91 580.18 -8.19%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1,935 2,093 +8.16%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 132.27 141.97 +7.33%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 16.92 15.81 -6.57%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 130.26 138.56 +6.37%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,903 3,077 +5.96%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 121.99 128.97 +5.72%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] 4,324 4,566 +5.60%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-None] 49.20 51.40 +4.48%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,034 2,900 -4.43%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 267.83 279.65 +4.41%
benchmarks/test_envs_benchmark.py::test_simple 1.8134 1.7339 -4.38%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None] 339.53 353.72 +4.18%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 22,268 23,163 +4.02%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] 331.95 345.13 +3.97%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] 265.22 275.74 +3.97%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] 19,061 19,815 +3.96%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[True-None] 222.44 231.10 +3.89%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 326.06 338.69 +3.88%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] 262.45 272.50 +3.83%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 684.78 709.98 +3.68%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 115.54 119.79 +3.68%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] 1,750 1,814 +3.61%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1,113 1,153 +3.57%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] 7,133 7,387 +3.56%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 411.92 426.43 +3.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] 38,266 39,587 +3.45%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False] 63,975 66,157 +3.41%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,804 1,865 +3.39%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[200-img_shape1-large_batch] 10.03 10.36 +3.36%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-None] 92.82 95.89 +3.30%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] 69.64 71.93 +3.28%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-True-0-gru] 1.4210 1.4676 +3.28%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 275.45 284.38 +3.24%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 8,189 7,930 -3.16%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 38,017 39,196 +3.10%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 5,024 4,869 -3.09%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] 42,824 44,147 +3.09%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 30,770 31,718 +3.08%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[reduce-overhead-None] 227.64 234.62 +3.07%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[reduce-overhead-None] 288.73 297.34 +2.98%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cpu] 51.56 53.09 +2.97%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 164.34 169.19 +2.95%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-4] 144.23 148.41 +2.90%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1] 264.64 272.26 +2.88%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] 31,714 30,821 -2.81%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] 32,849 33,772 +2.81%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 0.5938 0.6103 +2.78%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 28,557 29,349 +2.77%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] 19.94 19.41 -2.66%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 38,255 39,264 +2.64%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] 213.71 219.35 +2.64%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] 2,441 2,504 +2.59%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-backward] 32.94 33.79 +2.57%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 115.01 117.85 +2.46%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 165.08 169.12 +2.45%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] 465.59 476.91 +2.43%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 641.41 656.90 +2.42%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] 33,200 32,405 -2.39%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] 12,055 12,343 +2.39%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 34,497 35,304 +2.34%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] 19,782 20,245 +2.34%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 307.04 314.14 +2.31%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[False-backward] 55.52 56.74 +2.20%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 78,205 79,905 +2.17%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 503.56 514.51 +2.17%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 57,761 58,963 +2.08%
benchmarks/test_envs_benchmark.py::test_serial 0.5822 0.5943 +2.07%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.5245 0.5352 +2.04%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-backward] 28.07 28.63 +1.99%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] 3.1251 3.0638 -1.96%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 386.57 394.05 +1.94%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 42,994 43,808 +1.89%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 159.39 162.39 +1.88%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward] 242.81 247.26 +1.84%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 20,791 21,156 +1.76%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 288.47 293.45 +1.73%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] 24,200 24,612 +1.70%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru] 1.3315 1.3541 +1.70%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 287.66 292.47 +1.67%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 909.72 894.54 -1.67%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64] 1.2513 1.2717 +1.63%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] 133.98 136.16 +1.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 23.26 23.64 +1.60%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64] 3.0301 3.0781 +1.58%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] 371,841 366,077 -1.55%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 119.34 121.18 +1.55%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] 17.89 18.16 +1.51%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari] 29.07 29.50 +1.47%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] 2.0694 2.0392 -1.46%
... ... ... Showing 120 of 216 comparisons, sorted by absolute change.

GPU

Compared 226 benchmarks. Regressions over 5%: 6. Improvements over 5%: 11.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 44.04 196.95 +347.16%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 184.47 48.81 -73.54%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 76.30 103.11 +35.14%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 84.12 105.07 +24.90%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1,919 2,305 +20.14%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1,016 852.51 -16.09%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 433.95 486.38 +12.08%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 652.25 717.16 +9.95%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 477.91 515.35 +7.84%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,616 2,814 +7.58%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] 260.52 278.58 +6.93%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 44,232 41,388 -6.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1,977 1,857 -6.08%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 335.37 354.94 +5.84%
benchmarks/test_envs_benchmark.py::test_simple 1.2349 1.1628 -5.83%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,777 2,937 +5.76%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] 6,191 5,858 -5.38%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] 344.11 360.77 +4.84%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 557.85 531.36 -4.75%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1,961 2,054 +4.75%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 636.09 665.80 +4.67%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 863.40 825.35 -4.41%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,707 2,589 -4.37%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] 1,851 1,930 +4.26%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] 222.47 231.47 +4.05%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[100-img_shape0-atari] 20.51 19.75 -3.69%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 717.74 741.34 +3.29%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,715 2,804 +3.29%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 500.61 484.27 -3.26%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] 49,134 50,726 +3.24%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] 144.81 149.50 +3.24%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] 635.28 615.78 -3.07%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,817 1,873 +3.07%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 408.70 421.18 +3.05%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] 20.21 19.60 -3.03%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None] 381.40 392.82 +2.99%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-None] 615.42 633.84 +2.99%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 473.15 486.90 +2.91%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] 735.89 714.62 -2.89%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,794 2,713 -2.88%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 490.43 504.56 +2.88%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] 33,755 34,727 +2.88%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 343.60 353.49 +2.88%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64] 1.2605 1.2247 -2.84%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] 264.94 272.40 +2.82%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-1] 80.94 78.68 -2.78%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp... 1,500 1,462 -2.52%
benchmarks/test_envs_benchmark.py::test_serial 0.4148 0.4252 +2.52%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 699.07 716.63 +2.51%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] 18.02 17.57 -2.50%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 177.47 173.10 -2.46%
benchmarks/test_envs_benchmark.py::test_parallel 0.5362 0.5492 +2.43%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] 4.5295 4.4208 -2.40%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 52.70 51.45 -2.38%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 165.31 169.23 +2.37%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] 71.45 69.77 -2.36%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-memmap_cpu_storage_cud... 986.18 962.98 -2.35%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] 56,082 54,768 -2.34%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] 15.10 14.75 -2.32%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-4] 143.88 140.59 -2.29%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] 27,891 27,257 -2.27%
benchmarks/test_envs_benchmark.py::test_transformed 0.7071 0.6912 -2.26%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 140.05 136.89 -2.25%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] 23,266 23,776 +2.19%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 386.57 378.12 -2.19%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 21,469 21,931 +2.15%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 6.6916 6.5478 -2.15%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 87.42 89.24 +2.09%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] 7,019 6,872 -2.08%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,683 3,606 -2.08%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 21,050 20,614 -2.07%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 34,965 34,255 -2.03%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] 34,488 33,796 -2.01%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] 462.20 471.33 +1.98%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] 21.24 20.83 -1.97%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-backward] 126.18 128.65 +1.96%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] 40,684 41,477 +1.95%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False] 29,570 29,001 -1.93%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 194.49 190.80 -1.90%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 442.01 450.30 +1.88%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 51.47 50.51 -1.85%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] 190.16 193.67 +1.85%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-16] 5.0410 4.9504 -1.80%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 160.15 163.02 +1.79%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 23.31 22.89 -1.79%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[200-img_shape1-large_batch] 10.38 10.20 -1.78%
benchmarks/test_collectors_benchmark.py::test_async 11.00 10.81 -1.75%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 21.49 21.11 -1.75%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 43.18 42.43 -1.75%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-None] 226.26 222.38 -1.71%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 270.69 275.33 +1.71%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 22.57 22.19 -1.69%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[1000000-cuda] 2,177 2,213 +1.65%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1,249 1,228 -1.65%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 166.19 168.87 +1.61%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] 816.20 829.25 +1.60%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] 19,673 19,981 +1.56%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 166.84 169.41 +1.54%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] 22.49 22.15 -1.54%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] 12,087 11,902 -1.53%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] 29,620 30,065 +1.50%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 38,695 38,116 -1.50%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16] 12.21 12.03 -1.47%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] 80.47 79.30 -1.46%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 388.60 394.24 +1.45%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-False] 44,602 43,954 -1.45%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64] 3.0701 3.0261 -1.43%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] 98.20 99.55 +1.37%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1,011 998.01 -1.33%
benchmarks/test_collectors_benchmark.py::test_sync_pixels 10.06 10.19 +1.30%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 76,694 75,718 -1.27%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] 388.48 383.59 -1.26%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 306.73 310.52 +1.24%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 10.38 10.25 -1.23%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None] 329.87 333.82 +1.20%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 440.92 446.18 +1.19%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari] 17.11 17.30 +1.14%
benchmarks/test_collectors_benchmark.py::test_async_pixels 10.59 10.71 +1.14%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] 677.99 685.69 +1.14%
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] 19.81 19.59 -1.12%
... ... ... Showing 120 of 226 comparisons, sorted by absolute change.

vmoens added 3 commits June 22, 2026 09:18
@vmoens
Update
2e76fc2 
[ghstack-poisoned]
@vmoens
Update
51d07a4 
[ghstack-poisoned]
@vmoens
Update
6907fe4 
[ghstack-poisoned]
This was referenced Jun 24, 2026
Merged
Closed
theap06
theap06 approved these changes Jun 27, 2026
View reviewed changes

@theap06 theap06 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vmoens
Update
736f336 
[ghstack-poisoned]
This was referenced Jul 6, 2026
Open
Merged
Merged
@vmoens vmoens marked this pull request as ready for review July 6, 2026 08:59
vmoens added 2 commits July 6, 2026 10:04
@vmoens
Update
9cc4068 
[ghstack-poisoned]
@vmoens
Update
9430e2a 
[ghstack-poisoned]
This was referenced Jul 6, 2026
Merged
Merged
@vmoens
Update
8115361 
[ghstack-poisoned]
vmoens added 2 commits July 6, 2026 13:37
@vmoens
Update
9448937 
[ghstack-poisoned]
@vmoens
Update
40b95d0 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add remote policy module for inference server clients
307eb5a 
ghstack-source-id: c6f010b
Pull-Request: #3894
# Conflicts:
#	docs/source/reference/modules_inference_server.rst
@vmoens vmoens merged commit 40b95d0 into gh/vmoens/289/base Jul 6, 2026
108 of 109 checks passed
@vmoens vmoens deleted the gh/vmoens/289/head branch July 6, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@theap06 theap06 theap06 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Collectors Documentation Improvements or additions to documentation Feature New feature Integrations/torch_geometric Integrations Modules

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vmoens @theap06