Skip to content

[Feature] Add max-inflight guard for remote policy clients#3897

Merged
vmoens merged 14 commits into
gh/vmoens/292/basefrom
gh/vmoens/292/head
Jul 6, 2026
Merged

[Feature] Add max-inflight guard for remote policy clients#3897
vmoens merged 14 commits into
gh/vmoens/292/basefrom
gh/vmoens/292/head

Conversation

@vmoens

@vmoens vmoens commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@vmoens
Update
bf71148 
[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3897

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jun 21, 2026
Merged
Merged
Merged
Merged
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Results: PR 70713b7b vs main 8e4a3fb9

Benchmark run: https://github.com/pytorch/rl/actions/runs/28797327190

Higher ops/sec is better. Tables are sorted by largest absolute change.

CPU

Compared 216 benchmarks. Regressions over 5%: 13. Improvements over 5%: 26.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 50.34 194.54 +286.46%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 197.54 39.84 -79.83%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] 60.80 87.18 +43.39%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,777 3,692 +32.93%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,725 3,576 +31.23%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2,645 3,435 +29.89%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,755 3,461 +25.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,594 3,160 +21.85%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 28.37 33.80 +19.12%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,617 3,106 +18.65%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,875 3,364 +17.01%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 760.22 871.88 +14.69%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 20.50 23.47 +14.53%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1,871 2,140 +14.40%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 29.23 25.14 -14.00%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 858.33 976.70 +13.79%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1,079 934.69 -13.38%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 751.64 845.97 +12.55%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] 7.0639 6.1815 -12.49%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1,942 2,162 +11.33%
benchmarks/test_collectors_benchmark.py::test_single_with_rb 7.5752 6.7223 -11.26%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 220.46 242.33 +9.92%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 480.45 522.29 +8.71%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 55.29 59.87 +8.28%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] 34,794 32,193 -7.47%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 95.16 101.92 +7.10%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,083 2,228 +6.93%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] 10.54 9.8544 -6.51%
benchmarks/test_envs_benchmark.py::test_parallel 0.9365 0.8779 -6.26%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-None] 276.28 259.51 -6.07%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 7,338 7,752 +5.63%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 47.24 49.78 +5.39%
benchmarks/test_envs_benchmark.py::test_simple 1.8014 1.7086 -5.15%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 23.02 24.20 +5.14%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] 4,533 4,301 -5.13%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 108.58 114.09 +5.08%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 81,618 77,478 -5.07%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 51.31 53.90 +5.05%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,101 1,995 -5.05%
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] 37.51 39.35 +4.91%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 279.81 267.39 -4.44%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 270.25 281.77 +4.26%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 272.30 283.89 +4.26%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 137.18 131.37 -4.24%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 120.89 125.98 +4.21%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1,061 1,106 +4.18%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 20,779 19,912 -4.17%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 112.81 117.22 +3.91%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 397.48 412.26 +3.72%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 177.09 170.59 -3.67%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 21,515 20,729 -3.66%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 24.29 25.17 +3.62%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 60,724 58,537 -3.60%
benchmarks/test_envs_benchmark.py::test_transformed 0.8917 0.9238 +3.59%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-backward] 63.30 61.08 -3.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] 24,568 23,704 -3.52%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 707.18 731.19 +3.40%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] 1,714 1,772 +3.38%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 502.32 518.66 +3.25%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.5392 0.5217 -3.24%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] 52,152 50,512 -3.15%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None] 444.10 458.07 +3.14%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] 70.04 72.16 +3.04%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] 52,419 50,850 -2.99%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-None] 80.92 83.32 +2.96%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 40,667 39,472 -2.94%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[50-img_shape0-small] 3,584 3,481 -2.87%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-None] 706.72 687.14 -2.77%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 514.75 500.68 -2.73%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[reduce-overhead-None] 551.74 566.56 +2.69%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 29,361 28,613 -2.55%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] 18,618 18,154 -2.49%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 31,225 30,468 -2.42%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 176.63 172.41 -2.39%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 402.09 411.66 +2.38%
benchmarks/test_envs_benchmark.py::test_serial 0.5845 0.5984 +2.36%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] 15.22 14.86 -2.36%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 22,939 22,402 -2.34%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None] 348.51 340.37 -2.34%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] 442.96 453.21 +2.31%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] 616.04 630.29 +2.31%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 174.18 170.17 -2.31%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] 257.98 263.59 +2.18%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64] 1.2520 1.2254 -2.12%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 17.04 17.40 +2.10%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward] 244.25 239.18 -2.08%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-False] 0.2298 0.2252 -2.03%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 169.81 166.42 -2.00%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None] 122.98 120.55 -1.98%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] 158.33 161.41 +1.94%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] 117.62 115.34 -1.94%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 21,507 21,090 -1.94%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 140.80 143.50 +1.92%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] 43,281 42,469 -1.87%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] 4.4454 4.5263 +1.82%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] 35,993 35,342 -1.81%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 35,572 36,205 +1.78%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[reduce-overhead-None] 683.93 696.12 +1.78%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-1] 80.20 78.77 -1.78%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 328.26 333.98 +1.74%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 167.07 164.17 -1.74%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] 121.53 123.63 +1.73%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2,702 2,748 +1.71%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] 2.9639 2.9132 -1.71%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] 8.6915 8.8400 +1.71%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True] 1.3513 1.3282 -1.71%
benchmarks/test_collectors_benchmark.py::test_async 17.89 17.59 -1.69%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 53.95 54.85 +1.68%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru] 1.3644 1.3416 -1.67%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 637.45 648.10 +1.67%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] 541.54 550.50 +1.65%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] 59.07 60.03 +1.63%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-None] 49.06 49.82 +1.56%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 41.98 42.63 +1.54%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[False-None] 37.43 38.01 +1.53%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 559.30 550.83 -1.51%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-memmap_cpu_storage_cpu... 82.16 83.39 +1.50%
benchmarks/test_collectors_benchmark.py::test_sync 17.46 17.20 -1.49%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 702.62 712.94 +1.47%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] 190.89 193.58 +1.41%
... ... ... Showing 120 of 216 comparisons, sorted by absolute change.

GPU

Compared 226 benchmarks. Regressions over 5%: 21. Improvements over 5%: 11.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 44.72 195.36 +336.84%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,242 527.26 -76.49%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 183.41 50.96 -72.21%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 60.99 88.60 +45.25%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 76.78 103.19 +34.40%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,854 3,766 +31.99%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2,731 3,523 +28.98%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 3,697 2,761 -25.31%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 85.26 105.43 +23.66%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,457 2,666 -22.87%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,364 2,714 -19.32%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,364 2,715 -19.30%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,365 1,931 -18.36%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 1,032 893.55 -13.42%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 5.6204 4.9028 -12.77%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,289 2,000 -12.63%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] 801.64 710.51 -11.37%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,028 2,240 +10.45%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,568 2,836 +10.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 786.11 704.98 -10.32%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] 6.4378 7.0468 +9.46%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img] 586.16 544.82 -7.05%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 21,431 19,989 -6.73%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 38,149 35,744 -6.30%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] 33,456 31,348 -6.30%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] 19,999 18,741 -6.29%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] 50,863 47,823 -5.98%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 362.13 341.24 -5.77%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True] 42,397 39,956 -5.76%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,991 3,161 +5.69%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 469.73 443.92 -5.50%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 3,990 4,209 +5.48%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 11,882 12,461 +4.87%
benchmarks/test_envs_benchmark.py::test_parallel 0.5331 0.5588 +4.81%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] 9.0065 8.5733 -4.81%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 10.02 10.51 +4.81%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 530.22 506.29 -4.51%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 181.29 173.19 -4.46%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False] 32,463 31,083 -4.25%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True] 20,678 19,851 -4.00%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] 30,626 29,435 -3.89%
benchmarks/test_envs_benchmark.py::test_simple 1.2686 1.2197 -3.86%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True] 18,794 18,077 -3.82%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] 1,902 1,975 +3.81%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 337.54 324.69 -3.81%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] 56,132 54,049 -3.71%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[200-img_shape1-large_batch] 10.53 10.14 -3.70%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,882 1,951 +3.69%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 76,793 74,026 -3.60%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] 7,069 7,323 +3.59%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 495.84 513.43 +3.55%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] 155.45 149.99 -3.51%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 34,553 33,348 -3.49%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] 34,128 32,945 -3.47%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 30,066 29,035 -3.43%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] 4,957 4,788 -3.41%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 139.10 134.41 -3.37%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 661.39 639.55 -3.30%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 19,730 19,086 -3.26%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 57,597 55,726 -3.25%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 21,939 21,230 -3.23%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False] 29,346 28,421 -3.15%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4] 68.94 71.08 +3.11%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-triton-False-0-gru] 20.80 20.16 -3.06%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 28,659 27,793 -3.02%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16] 17.51 18.04 +3.01%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 42,372 41,103 -2.99%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 263.33 271.15 +2.97%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False] 64,171 62,310 -2.90%
benchmarks/test_envs_benchmark.py::test_transformed 0.7260 0.7051 -2.87%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cpu_sampler] 89.88 87.30 -2.87%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False] 31,619 30,728 -2.82%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False] 35,089 34,124 -2.75%
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] 12.51 12.84 +2.64%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 412.10 422.98 +2.64%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-gru] 22.94 23.54 +2.60%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 708.93 690.69 -2.57%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 733.92 752.76 +2.57%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] 50.00 51.26 +2.51%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True] 17,940 17,506 -2.42%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] 23,215 22,671 -2.34%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[50-img_shape0-small] 885.66 865.13 -2.32%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari] 30.82 30.11 -2.30%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 53.64 52.42 -2.27%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 169.98 166.24 -2.20%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 353.44 360.79 +2.08%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 507.52 518.03 +2.07%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] 38,259 37,502 -1.98%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] 287.36 281.85 -1.91%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 50.28 49.32 -1.91%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari] 17.75 17.43 -1.85%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] 627.31 638.78 +1.83%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] 4,890 4,979 +1.82%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] 83.20 81.69 -1.80%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 41.19 41.93 +1.79%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 397.95 404.88 +1.74%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 48.16 48.99 +1.73%
benchmarks/test_envs_benchmark.py::test_serial 0.4297 0.4371 +1.72%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 280.55 275.73 -1.72%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] 27,001 26,542 -1.70%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None] 116.11 114.20 -1.64%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1] 475.92 483.54 +1.60%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 330.57 325.38 -1.57%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-4] 187.39 184.48 -1.55%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 22.11 21.77 -1.55%
benchmarks/test_collectors_benchmark.py::test_single_with_rb_pixels 5.3227 5.4033 +1.52%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16] 11.95 12.13 +1.51%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 189.63 192.48 +1.50%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64] 4.4396 4.5051 +1.47%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 38,044 37,484 -1.47%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-memmap_cpu_storage_cud... 994.23 1,009 +1.45%
benchmarks/test_collectors_benchmark.py::test_sync_pixels 10.15 10.30 +1.41%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-16] 4.9674 5.0367 +1.39%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm] 21.83 21.53 -1.39%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] 740.77 751.04 +1.39%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 52.94 52.22 -1.36%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1,298 1,315 +1.36%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None] 770.73 760.34 -1.35%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 910.74 898.51 -1.34%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] 838.08 849.27 +1.34%
... ... ... Showing 120 of 226 comparisons, sorted by absolute change.

@vmoens
Update
581ce7e 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jun 22, 2026
@vmoens
[Feature] Add max-inflight guard for remote policy clients
4414259 
ghstack-source-id: 4123ded
Pull-Request: #3897
@vmoens
Update
f1af2b4 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jun 24, 2026
@vmoens
[Feature] Add max-inflight guard for remote policy clients
9d84e0e 
ghstack-source-id: 6384e65
Pull-Request: #3897
@vmoens
Update
7653b7d 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jun 24, 2026
@vmoens
[Feature] Add max-inflight guard for remote policy clients
2a2b9b0 
ghstack-source-id: c230dc7
Pull-Request: #3897
This was referenced Jun 24, 2026
Merged
Closed
theap06
theap06 suggested changes Jun 26, 2026
View reviewed changes

@theap06 theap06 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense overall. my one gripe is the result() call. Releasing the slot only when result() is called works for the synchronous collector path, but it leaks a slot if a future gets dropped, and a result() timeout frees the slot while the request is still running. I'd just release on the future completing add_done_callback instead.

also, for the getattr function, since it forwards to self._future, it'll infinite-loop if _future ever isn't set yet. Easy fix is to forego the private names.

@vmoens
Update
c536fe9 
[ghstack-poisoned]
This was referenced Jul 6, 2026
Open
Merged
Merged
@vmoens vmoens marked this pull request as ready for review July 6, 2026 08:59
@vmoens
Update
00118e2 
[ghstack-poisoned]
@vmoens
Update
1a450ef 
[ghstack-poisoned]
This was referenced Jul 6, 2026
Merged
Merged
vmoens added 7 commits July 6, 2026 12:56
@vmoens
Update
069cab9 
[ghstack-poisoned]
@vmoens
Update
dc61ce2 
[ghstack-poisoned]
@vmoens
Update
c2f24d4 
[ghstack-poisoned]
@vmoens
Update
0431f3b 
[ghstack-poisoned]
@vmoens
Update
066a8a7 
[ghstack-poisoned]
@vmoens
Update
7fefac5 
[ghstack-poisoned]
@vmoens
Update
70713b7 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add max-inflight guard for remote policy clients
539aa07 
ghstack-source-id: 4e26907
Pull-Request: #3897
@vmoens vmoens merged commit 70713b7 into gh/vmoens/292/base Jul 6, 2026
103 of 104 checks passed
@vmoens vmoens deleted the gh/vmoens/292/head branch July 6, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@theap06 theap06 theap06 requested changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Collectors Feature New feature Integrations/torch_geometric Integrations Modules

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vmoens @theap06