[Feature] Add async VLA GRPO collection modes

[vmoens]

Stack from ghstack (oldest at bottom):

• [BugFix] Stabilize LIBERO VLA GRPO training #3940
• -> [Feature] Add async VLA GRPO collection modes #3909
• [Feature] Add max-inflight guard for remote policy clients #3897
• [Feature] Add process inference server control plane #3896
• [Feature] Track behavior policy versions in inference server #3895
• [Feature] Add remote policy module for inference server clients #3894
• [Refactor] Add structured inference server config objects #3893
• [Feature] Add async policy-server collector benchmark prototype #3939
• [Feature] Add _Contiguous module and reshape improvements to encoders/decoders #3306

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3909

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 27 Pending

As of commit 4c8d654 with merge base 6680c17 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

Benchmark Results: PR 4c8d654b vs main a7a7e954

Benchmark run: https://github.com/pytorch/rl/actions/runs/28780236630

Higher ops/sec is better. Tables are sorted by largest absolute change.

CPU

Compared 216 benchmarks. Regressions over 5%: 43. Improvements over 5%: 28.

Benchmark	main ops	PR ops	Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	35.84	196.40	+447.97%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	2,080	453.57	-78.19%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	30.22	53.39	+76.64%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	185.53	53.03	-71.42%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False]	54.07	86.27	+59.58%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True]	54.62	86.93	+59.17%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td_lambda_return_estimate-True-False]	55.05	86.73	+57.55%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1]	531.31	645.77	+21.54%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64]	10.49	8.2324	-21.53%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16]	35.87	28.44	-20.71%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward]	813.68	968.11	+18.98%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	28.81	33.88	+17.61%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-1]	69.42	81.10	+16.82%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same]	28.29	23.63	-16.48%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64]	6.9877	5.9897	-14.28%
benchmarks/test_envs_benchmark.py::test_parallel	0.8316	0.9502	+14.27%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward]	366.80	414.63	+13.04%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same]	19.68	22.19	+12.76%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3,264	2,867	-12.16%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3,291	3,674	+11.63%
benchmarks/test_collectors_benchmark.py::test_single_with_rb	7.2799	6.4404	-11.53%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64]	12.03	10.67	-11.34%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3,125	2,784	-10.91%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2,184	1,964	-10.09%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2,184	1,966	-9.95%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward]	126.16	138.28	+9.61%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-True]	19,003	17,235	-9.30%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4]	18.69	20.39	+9.06%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-False]	45,831	41,797	-8.80%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3,610	3,299	-8.62%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True]	19,891	18,206	-8.47%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-True-False]	29,683	27,184	-8.42%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True]	90.22	97.36	+7.92%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True]	21,211	19,589	-7.64%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False]	27,296	25,239	-7.54%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False]	50,537	46,741	-7.51%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-True]	18,155	16,814	-7.39%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True]	28,776	26,677	-7.29%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True]	31,340	29,062	-7.27%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	514.66	551.20	+7.10%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True]	22,097	20,607	-6.75%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-True]	21,218	19,795	-6.71%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-False]	32,149	29,995	-6.70%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False]	55,025	51,362	-6.66%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3,575	3,337	-6.63%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	2,891	3,078	+6.47%
benchmarks/test_envs_benchmark.py::test_simple	1.8015	1.6885	-6.28%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False]	38,493	36,078	-6.27%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari]	262.66	279.09	+6.25%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True]	30,427	28,541	-6.20%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False]	49,978	46,889	-6.18%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True]	35,177	33,009	-6.16%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-False]	31,817	29,890	-6.06%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False]	63,962	60,093	-6.05%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64]	1.2067	1.2795	+6.03%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True]	19,933	18,762	-5.88%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True]	24,452	23,021	-5.85%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-16]	4.8195	5.0987	+5.79%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False]	7,075	7,480	+5.73%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy]	354,491	334,395	-5.67%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-lstm]	0.8928	0.8428	-5.60%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-64]	4.3414	4.5825	+5.55%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1,078	1,019	-5.47%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False]	77,390	73,161	-5.46%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-False]	64,310	60,803	-5.45%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-True]	41,902	39,661	-5.35%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	24.54	25.83	+5.27%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm]	1.9513	2.0531	+5.22%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-4]	69.08	72.61	+5.11%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img]	385.44	404.93	+5.06%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-16]	17.33	18.20	+5.04%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True]	0.5257	0.4998	-4.94%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage]	7.8714	8.2583	+4.91%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False]	56,490	53,794	-4.77%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512]	111.14	105.86	-4.75%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-4]	47.16	49.35	+4.64%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True]	38,071	36,304	-4.64%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[50-img_shape0-small]	3,339	3,494	+4.63%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True]	37,530	35,862	-4.44%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img]	405.35	423.21	+4.41%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64]	2.9646	3.0935	+4.35%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16]	11.81	12.32	+4.32%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save]	6,836	7,130	+4.30%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2,945	3,067	+4.13%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru]	1.3713	1.3149	-4.12%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img]	176.61	169.95	-3.77%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward]	109.53	113.63	+3.75%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant]	4,226	4,070	-3.70%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	2,247	2,165	-3.64%
benchmarks/test_envs_benchmark.py::test_transformed	0.9039	0.8712	-3.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	191.30	198.19	+3.60%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False]	34,097	32,871	-3.60%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari]	28.52	29.54	+3.58%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16]	41.76	43.21	+3.48%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	1,107	1,069	-3.40%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1]	262.51	271.39	+3.38%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False]	33,782	32,703	-3.20%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None]	441.59	455.67	+3.19%
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False]	37.07	35.90	-3.18%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-True-0-gru]	1.4717	1.4269	-3.04%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[reduce-overhead-None]	475.22	460.95	-3.00%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari]	4,980	4,832	-2.97%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False]	42,410	41,180	-2.90%
benchmarks/test_collectors_benchmark.py::test_single	8.8713	8.6245	-2.78%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-16]	47.64	48.95	+2.75%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle]	12,417	12,111	-2.46%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None]	1,705	1,746	+2.43%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch]	308.30	315.80	+2.43%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward]	114.64	117.38	+2.39%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward]	517.89	505.75	-2.35%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None]	82.64	84.50	+2.25%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True]	1.2728	1.3009	+2.21%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward]	91.70	89.73	-2.16%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	161.46	164.94	+2.16%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None]	325.88	332.88	+2.15%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True]	32,396	31,705	-2.13%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None]	343.56	336.24	-2.13%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[reduce-overhead-None]	217.73	222.28	+2.09%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1]	470.79	480.46	+2.05%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward]	58.74	59.94	+2.03%
...	...	...	Showing 120 of 216 comparisons, sorted by absolute change.

GPU

Compared 226 benchmarks. Regressions over 5%: 6. Improvements over 5%: 15.

Benchmark	main ops	PR ops	Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	44.32	194.38	+338.56%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None]	76.57	102.47	+33.83%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2,534	3,360	+32.61%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None]	84.90	106.13	+25.00%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	2,728	3,168	+16.13%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3,525	3,035	-13.91%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2,577	2,931	+13.72%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True]	484.13	547.53	+13.10%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	2,207	1,950	-11.63%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	834.70	771.50	-7.57%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	2,864	3,074	+7.32%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1]	552.70	591.24	+6.97%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	2,854	3,050	+6.88%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari]	3,948	4,213	+6.73%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch]	142.43	132.98	-6.63%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2,712	2,887	+6.45%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[reduce-overhead-None]	816.14	866.09	+6.12%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2,082	2,199	+5.60%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari]	647.28	611.12	-5.59%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True]	46.53	48.96	+5.24%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2,030	1,928	-5.04%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512]	49.58	47.12	-4.97%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None]	1,832	1,923	+4.97%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True]	20,508	21,522	+4.95%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward]	354.69	338.50	-4.56%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	1,355	1,415	+4.42%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True]	30,049	31,239	+3.96%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	442.04	425.04	-3.84%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant]	4,728	4,909	+3.83%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari]	689.96	664.14	-3.74%
benchmarks/test_collectors_benchmark.py::test_sync_pixels	9.8965	10.26	+3.72%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage]	8.4084	8.7176	+3.68%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None]	609.07	629.68	+3.38%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None]	283.33	273.77	-3.38%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm]	77.69	75.08	-3.37%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch]	309.53	299.09	-3.37%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False]	0.5882	0.6080	+3.37%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	189.57	195.73	+3.25%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors]	22,906	23,646	+3.23%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None]	725.35	748.18	+3.15%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-1]	461.78	447.35	-3.12%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward]	359.69	348.49	-3.11%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False]	11,503	11,856	+3.06%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3,706	3,593	-3.04%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	21.52	22.15	+2.97%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img]	406.56	394.64	-2.93%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	987.37	958.51	-2.92%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	47.53	48.91	+2.91%
benchmarks/test_envs_benchmark.py::test_simple	1.2408	1.2065	-2.76%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	481.43	494.50	+2.72%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True]	28,098	28,855	+2.69%
benchmarks/test_collectors_benchmark.py::test_async	10.82	11.11	+2.68%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb_cuda[200-img_shape1-large_batch]	6.8040	6.6254	-2.63%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch]	15.54	15.13	-2.62%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-False]	0.2199	0.2256	+2.58%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img]	175.49	170.95	-2.58%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant]	4,693	4,814	+2.57%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-16]	5.0680	4.9392	-2.54%
benchmarks/test_envs_benchmark.py::test_transformed	0.7170	0.6993	-2.46%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16]	36.12	35.23	-2.46%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[100-img_shape0-atari]	30.63	29.89	-2.43%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward]	222.23	227.63	+2.43%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[200-img_shape1-large_batch]	10.54	10.28	-2.43%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True]	37,702	38,613	+2.41%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False]	51,007	49,810	-2.35%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-backward]	133.58	130.47	-2.33%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[100-img_shape0-atari]	20.81	20.33	-2.33%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch]	744.38	727.08	-2.32%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None]	1,903	1,947	+2.30%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy]	374,593	366,047	-2.28%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	53.70	52.48	-2.27%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16]	42.69	41.72	-2.27%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64]	3.0676	2.9994	-2.22%
benchmarks/test_envs_benchmark.py::test_parallel	0.5400	0.5517	+2.16%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True]	33,976	34,707	+2.15%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	2,011	1,968	-2.11%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-False]	34,648	35,376	+2.10%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None]	121.33	118.80	-2.08%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False]	63,270	64,583	+2.07%
benchmarks/test_envs_benchmark.py::test_serial	0.4246	0.4334	+2.07%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-64]	1.2593	1.2333	-2.06%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small]	4,317	4,405	+2.05%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[200-img_shape1-large_batch]	8.8780	8.6984	-2.02%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False]	55,156	56,269	+2.02%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-None]	775.77	760.14	-2.01%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True]	19,458	19,846	+2.00%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	23.62	23.15	-1.99%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb_cuda[100-img_shape0-atari]	13.57	13.30	-1.99%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-224-224-1]	272.16	277.43	+1.94%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape2-large_img]	554.64	544.08	-1.90%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward]	350.23	356.82	+1.88%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp...	1,543	1,514	-1.86%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch]	325.87	319.96	-1.82%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari]	17.83	17.52	-1.74%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	1,351	1,328	-1.71%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-False]	44,635	45,364	+1.63%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None]	115.06	113.18	-1.63%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True]	19,719	20,038	+1.62%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	53.88	53.03	-1.59%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-False-0-lstm]	22.00	21.68	-1.44%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward]	151.57	149.40	-1.43%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward]	275.53	279.43	+1.41%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward]	82.95	81.81	-1.36%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None]	746.31	736.14	-1.36%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	826.88	815.76	-1.34%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward]	446.76	452.63	+1.31%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True]	32,547	32,965	+1.28%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	166.77	168.88	+1.26%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img]	418.36	413.22	-1.23%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-backward]	237.33	234.45	-1.21%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None]	844.40	854.65	+1.21%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[False-None]	341.42	345.56	+1.21%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64]	12.36	12.51	+1.20%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-None]	228.61	231.29	+1.17%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	792.33	783.42	-1.13%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-None]	406.29	401.79	-1.11%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward]	461.43	466.47	+1.09%
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False]	12.64	12.50	-1.07%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True]	37,941	37,543	-1.05%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False]	50,269	50,778	+1.01%
...	...	...	Showing 120 of 226 comparisons, sorted by absolute change.

[theap06]

LGTM left a few nitpicking comments regarding design

[vmoens]

Superseded: the async collection modes were reworked and absorbed into #3940, with the interaction-type propagation split out into #3941, so this intermediate implementation is no longer part of the stack.