Skip to content

[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity tooling#3942

Merged
vmoens merged 8 commits into
gh/vmoens/308/basefrom
gh/vmoens/308/head
Jul 6, 2026
Merged

[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity tooling#3942
vmoens merged 8 commits into
gh/vmoens/308/basefrom
gh/vmoens/308/head

Conversation

@vmoens

@vmoens vmoens commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Stack from ghstack (oldest at bottom):

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.

@vmoens
Update
bb9317f 
[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3942

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions github-actions Bot added Feature New feature sota-implementations/ and removed Feature New feature labels Jul 6, 2026
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 6, 2026
This was referenced Jul 6, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
@github-actions github-actions Bot added the Feature New feature label Jul 6, 2026
@vmoens
Update
aa809e6 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
7db0f52 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.


ghstack-source-id: 614dc3b
Pull-Request: #3942
@vmoens
Update
5c4a644 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
9b75618 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.


ghstack-source-id: 2b3fd4a
Pull-Request: #3942
@vmoens
Update
bc6c76d 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
0ada510 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.


ghstack-source-id: fd1aa41
Pull-Request: #3942
@vmoens
Update
6934359 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
4957880 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.


ghstack-source-id: 7a9c9f2
Pull-Request: #3942
@vmoens
Update
502ef04 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
752ad98 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.

ghstack-source-id: 204a5e0
Pull-Request: #3942
@github-actions

github-actions Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark Results: PR 5d5e35e6 vs main 3ce68bda

Benchmark run: https://github.com/pytorch/rl/actions/runs/28797323578

Higher ops/sec is better. Tables are sorted by largest absolute change.

CPU

Compared 216 benchmarks. Regressions over 5%: 8. Improvements over 5%: 23.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 49.20 191.24 +288.68%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 195.32 39.59 -79.73%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 54.98 87.66 +59.46%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td1_return_estimate-False-False] 55.32 87.54 +58.26%
benchmarks/test_objectives_benchmarks.py::test_values[vec_td_lambda_return_estimate-True-False] 55.28 86.97 +57.31%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 28.52 43.28 +51.75%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[False-None] 60.05 89.50 +49.03%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-16] 28.71 36.50 +27.14%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 231.51 284.95 +23.08%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 20.05 23.63 +17.89%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 29.53 25.27 -14.42%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-64] 9.3735 10.64 +13.55%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2,932 3,274 +11.68%
benchmarks/test_collectors_benchmark.py::test_single_with_rb 7.3781 6.5444 -11.30%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,244 2,005 -10.65%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 470.44 519.76 +10.49%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,988 3,289 +10.08%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,817 3,090 +9.70%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,339 3,018 -9.61%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-backward] 898.69 984.87 +9.59%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 5,007 5,468 +9.22%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 93.83 101.76 +8.45%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] 22,916 24,482 +6.83%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 793.69 741.03 -6.64%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,032 3,230 +6.53%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-64] 11.51 12.26 +6.46%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 924.58 865.10 -6.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 748.21 700.12 -6.43%
benchmarks/test_objectives_benchmarks.py::test_values[td0_return_estimate-False-False] 7,443 7,880 +5.88%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 264.80 280.01 +5.74%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 233.43 245.98 +5.38%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 403.89 423.54 +4.87%
benchmarks/test_envs_benchmark.py::test_simple 1.8050 1.7173 -4.86%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[200-img_shape3-large_batch] 136.27 142.70 +4.72%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 3,379 3,234 -4.29%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,180 3,044 -4.25%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[True-None] 1,699 1,770 +4.17%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] 187.24 194.99 +4.14%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[reduce-overhead-None] 256.47 266.86 +4.05%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 389.64 405.37 +4.04%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-False] 48,077 49,977 +3.95%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-False] 66,342 63,731 -3.94%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-single-True] 1.3166 1.3681 +3.92%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 406.32 421.16 +3.65%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-None] 253.43 262.43 +3.55%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 29,759 30,792 +3.47%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] 50,153 48,426 -3.44%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-True] 32,641 33,706 +3.26%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,210 3,105 -3.26%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 132.03 136.25 +3.20%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 163.95 169.09 +3.14%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] 6.2954 6.4869 +3.04%
benchmarks/test_objectives_benchmarks.py::test_redq_speed[True-None] 217.56 223.99 +2.95%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,807 1,859 +2.87%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] 2,489 2,560 +2.85%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 178.24 173.18 -2.84%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 25.38 24.67 -2.80%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb[200-img_shape1-large_batch] 14.82 15.23 +2.75%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sampler_sample_scale[10000000-cpu] 52.10 53.51 +2.70%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_with_rb[100-img_shape0-atari] 19.94 20.47 +2.68%
benchmarks/test_envs_benchmark.py::test_parallel 0.9875 0.9614 -2.64%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 278.06 285.30 +2.60%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 157.58 161.63 +2.57%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 123.55 126.72 +2.56%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 647.42 664.01 +2.56%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 28,114 28,824 +2.53%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.5082 0.5209 +2.51%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] 548.99 562.15 +2.40%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] 59.18 57.76 -2.39%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-False] 38,092 38,988 +2.35%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 328.99 336.60 +2.31%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 21,711 22,201 +2.26%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] 4,196 4,290 +2.25%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 160.88 164.49 +2.25%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-None] 49.63 50.69 +2.15%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 22.93 23.42 +2.13%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] 7,100 7,250 +2.12%
benchmarks/test_collectors_benchmark.py::test_sync 16.43 16.78 +2.10%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2,037 2,079 +2.02%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,124 2,082 -2.01%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[50-img_shape0-small] 7,122 7,262 +1.96%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-False-0-gru] 1.3130 1.3380 +1.90%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-None] 122.29 124.60 +1.89%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 274.47 279.65 +1.89%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] 332.69 338.97 +1.89%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-1] 627.34 638.88 +1.84%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 53.17 52.20 -1.83%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-True-True] 19,433 19,781 +1.79%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 513.90 522.84 +1.74%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 310.36 315.72 +1.73%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-64] 3.0604 3.0085 -1.70%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-4] 144.50 146.91 +1.67%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] 119.09 121.04 +1.63%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-None] 176.48 179.34 +1.62%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] 458.83 466.23 +1.61%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[False-backward] 88.61 90.03 +1.60%
benchmarks/test_envs_benchmark.py::test_serial 0.5830 0.5923 +1.59%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] 37,901 38,491 +1.56%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 690.36 701.01 +1.54%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 16.44 16.70 +1.53%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 58,126 57,242 -1.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 34,938 34,414 -1.50%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 20,370 20,674 +1.49%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-cudnn-True-0-gru] 1.4278 1.4491 +1.49%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 1,105 1,088 -1.49%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-False] 44,084 44,735 +1.48%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-False-False] 75,910 76,994 +1.43%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] 3.1313 3.1755 +1.41%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 2,162 2,192 +1.41%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-4] 162.72 165.00 +1.40%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 116.85 118.45 +1.37%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[False-backward] 516.28 523.38 +1.37%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] 12,254 12,086 -1.37%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] 20.00 20.27 +1.36%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 37,955 38,468 +1.35%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 162.98 165.11 +1.31%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-no-buffers-True] 0.2110 0.2138 +1.30%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-None] 269.62 273.03 +1.26%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 166.22 168.19 +1.18%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[reduce-overhead-None] 718.42 726.91 +1.18%
... ... ... Showing 120 of 216 comparisons, sorted by absolute change.

GPU

Compared 226 benchmarks. Regressions over 5%: 9. Improvements over 5%: 17.

Benchmark main ops PR ops Change
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 44.78 193.03 +331.10%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[reduce-overhead-None] 60.31 88.45 +46.66%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[reduce-overhead-None] 77.37 102.59 +32.61%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 3,329 2,554 -23.29%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[reduce-overhead-None] 84.92 104.30 +22.81%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 686.04 823.32 +20.01%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,655 3,154 +18.79%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2,497 2,961 +18.60%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[100-img_shape1-atari] 3,587 4,161 +16.01%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 3,149 2,709 -13.95%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-480-640-64] 6.4094 7.0138 +9.43%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,998 2,736 -8.74%
benchmarks/test_objectives_benchmarks.py::test_values[vec_generalized_advantage_estimate-True-True] 521.27 478.38 -8.23%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 716.00 771.61 +7.77%
benchmarks/test_collectors_benchmark.py::test_sync_preempt 9.6588 10.40 +7.64%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-False] 0.5585 0.5979 +7.05%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 2,871 2,672 -6.90%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_contiguous[200-img_shape3-large_batch] 737.92 687.55 -6.83%
benchmarks/test_objectives_benchmarks.py::test_dqn_speed[reduce-overhead-None] 1,801 1,918 +6.49%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[generalized_advantage_estimate-False-1-512] 48.14 45.05 -6.41%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 21.93 23.25 +6.05%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape2-large_img] 380.95 403.46 +5.91%
benchmarks/test_objectives_benchmarks.py::test_redq_deprec_speed[True-backward] 259.96 275.02 +5.79%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 22.36 23.60 +5.55%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2,794 2,647 -5.28%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 2,967 2,813 -5.20%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1,906 1,999 +4.89%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-gru] 49.03 46.64 -4.88%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-backward] 313.92 329.22 +4.87%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 958.01 1,002 +4.62%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[True-backward] 341.39 356.78 +4.51%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 49.48 51.69 +4.47%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-scan-True-0-lstm] 76.21 72.92 -4.32%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1,965 2,049 +4.26%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape2-large_img] 397.03 413.20 +4.07%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[True-backward] 351.94 337.68 -4.05%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-constant] 4,480 4,654 +3.88%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-memmap_cpu_storage_cud... 942.91 977.38 +3.66%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-None] 731.60 705.35 -3.59%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-backward] 356.56 343.97 -3.53%
benchmarks/test_objectives_benchmarks.py::test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 1,361 1,313 -3.52%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[100-img_shape1-atari] 675.61 652.35 -3.44%
benchmarks/test_envs_benchmark.py::test_parallel 0.5358 0.5538 +3.35%
benchmarks/test_envs_benchmark.py::test_simple 1.2359 1.1949 -3.31%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-4] 48.50 46.90 -3.31%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 21.60 22.31 +3.27%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[50-img_shape0-small] 4,209 4,345 +3.24%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 467.09 481.88 +3.17%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-False-True] 37,358 38,540 +3.16%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[False-backward] 265.78 274.17 +3.16%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[numpy] 346,505 357,073 +3.05%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-True-True] 20,111 20,698 +2.92%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 975.50 1,004 +2.91%
benchmarks/test_objectives_benchmarks.py::test_values[td1_return_estimate-False-False] 19.73 19.17 -2.84%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-True-True-False] 34,238 35,166 +2.71%
benchmarks/test_objectives_benchmarks.py::test_values[td_lambda_return_estimate-True-False] 11.99 11.67 -2.71%
benchmarks/test_rnn_reset_backends_benchmark.py::test_rnn_rollout_with_intermediate_resets[b256-t128-i32-h512-triton-False-0-gru] 20.04 20.58 +2.67%
benchmarks/test_replaybuffer_benchmark.py::test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 50.60 51.91 +2.60%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 827.58 848.85 +2.57%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-4] 19.31 19.80 +2.57%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[True-None] 727.12 745.20 +2.49%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[safetensors] 24,353 23,755 -2.46%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-True] 23,828 23,249 -2.43%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-False-True] 30,148 30,879 +2.42%
benchmarks/test_objectives_benchmarks.py::test_values[generalized_advantage_estimate-True-True] 47.37 46.24 -2.38%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-None] 802.56 821.64 +2.38%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[torch.save] 6,997 6,831 -2.37%
benchmarks/test_objectives_benchmarks.py::test_cql_speed[True-backward] 220.73 225.96 +2.37%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-16] 12.16 11.88 -2.37%
benchmarks/test_replaybuffer_benchmark.py::test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 167.03 170.84 +2.28%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-256-256-1] 190.42 194.72 +2.26%
benchmarks/test_envs_benchmark.py::test_serial 0.4189 0.4276 +2.09%
benchmarks/test_collectors_benchmark.py::test_sync 10.22 10.43 +2.08%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[pil-480-640-16] 4.9849 4.8818 -2.07%
benchmarks/test_objectives_benchmarks.py::test_reinforce_speed[reduce-overhead-None] 121.05 118.58 -2.04%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 184.71 188.40 +2.00%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-backward] 233.40 238.05 +1.99%
benchmarks/test_replaybuffer_benchmark.py::TestPrioritizedReplayBufferBenchmark::test_sample_mixed_devices[1000000-cuda_storage_cuda_samp... 1,466 1,495 +1.99%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[False-backward] 147.81 144.88 -1.98%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[pickle] 11,977 11,751 -1.89%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-1] 508.91 499.32 -1.89%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-256-256-16] 41.65 40.87 -1.87%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-constant] 4,649 4,735 +1.86%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-False-True-True] 19,542 19,900 +1.83%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[True-backward] 386.96 393.94 +1.80%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[100-img_shape1-atari] 624.87 635.93 +1.77%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[100-img_shape0-atari] 17.24 16.94 -1.74%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[16-same] 5.3652 5.4572 +1.71%
benchmarks/test_vla_preprocessing_benchmark.py::test_openvla_preprocessing_throughput[torchvision-224-224-4] 179.76 182.83 +1.71%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[True-None] 500.73 509.25 +1.70%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-True-True-False-False] 57,501 56,532 -1.69%
benchmarks/test_envs_benchmark.py::test_cat_frames_functional[4-same] 6.4995 6.6079 +1.67%
benchmarks/test_objectives_benchmarks.py::test_ppo_speed[False-None] 223.31 227.01 +1.66%
benchmarks/test_envs_benchmark.py::test_transformed 0.7060 0.6943 -1.65%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-True-True-False] 41,752 42,429 +1.62%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape2-large_img] 166.66 169.34 +1.61%
benchmarks/test_non_tensor_env_benchmark.py::test_non_tensor_env_rollout_speed[1000-parallel-buffers-True] 0.5129 0.5210 +1.59%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-False] 50,515 49,713 -1.59%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-None] 96.24 97.72 +1.54%
benchmarks/test_objectives_benchmarks.py::test_iql_speed[False-backward] 66.48 67.50 +1.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-True-False] 27,511 27,093 -1.52%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-True-False-True] 30,768 30,311 -1.48%
benchmarks/test_compressed_storage_benchmark.py::TestCompressedStorageBenchmark::test_tensor_to_bytestream_speed[untyped_storage] 7.9983 7.8805 -1.47%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[False-False-False-False-True] 28,717 28,299 -1.46%
benchmarks/test_collectors_benchmark.py::test_single_pixels 6.2516 6.1677 -1.34%
benchmarks/test_objectives_benchmarks.py::test_sac_speed[True-None] 613.25 621.44 +1.34%
benchmarks/test_replaybuffer_benchmark.py::test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 161.28 163.41 +1.32%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-None] 110.49 109.03 -1.32%
benchmarks/test_objectives_benchmarks.py::test_a2c_speed[reduce-overhead-None] 852.29 863.18 +1.28%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-True] 33,516 33,935 +1.25%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_lazystack_then_write[200-img_shape3-large_batch] 299.45 303.16 +1.24%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-False-True] 37,537 37,996 +1.22%
benchmarks/test_objectives_benchmarks.py::test_td3_speed[False-backward] 80.55 79.59 -1.19%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-True-False-True-True] 21,615 21,872 +1.19%
benchmarks/test_objectives_benchmarks.py::test_ddpg_speed[True-backward] 441.12 446.33 +1.18%
benchmarks/test_replaybuffer_benchmark.py::test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 420.18 415.29 -1.16%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_storage_write_lazystack[200-img_shape3-large_batch] 316.18 319.82 +1.15%
benchmarks/test_storage_write_benchmark.py::TestStorageWriteBenchmark::test_collector_stack_then_write[100-img_shape1-atari] 271.08 274.17 +1.14%
benchmarks/test_storage_write_benchmark.py::TestCollectorIntegrationBenchmark::test_collector_without_rb_cuda[200-img_shape1-large_batch] 8.5169 8.4202 -1.13%
benchmarks/test_envs_benchmark.py::test_step_mdp_speed[True-False-False-False-False] 53,860 54,454 +1.10%
... ... ... Showing 120 of 226 comparisons, sorted by absolute change.

@vmoens
Update
6242494 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
882784b 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.

ghstack-source-id: c3324d9
Pull-Request: #3942
@vmoens
Update
5d5e35e 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
e28c365 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.

ghstack-source-id: 0ee32ea
Pull-Request: #3942
vmoens added a commit that referenced this pull request Jul 6, 2026
@vmoens
[Feature] Add OpenVLA-OFT L1 reference wrapper and SimpleVLA parity t…
e0651a5 
…ooling

OpenVLAOFTL1Wrapper serves the official continuous L1-head OpenVLA-OFT
checkpoints (released action head + proprio projector, wrist image, 8-D
proprio) for deterministic reference and evaluation rollouts; token GRPO
training semantics are unchanged and the PPO loss still requires token
log-probabilities.

compare_simplevla_rollouts.py rolls the same (task, init-state) trajectories
through the TorchRL stack and the SimpleVLA-RL/VeRL reference and writes a
JSON parity report of per-trajectory token/action/success agreement.

The mode-selection plumbing (policy.mode=l1) ships in the commit below; on
its own that mode fails with an ImportError until this commit is applied.

ghstack-source-id: 0ee32ea
Pull-Request: #3942
@vmoens vmoens merged commit 5d5e35e into gh/vmoens/308/base Jul 6, 2026
101 of 105 checks passed
@vmoens vmoens deleted the gh/vmoens/308/head branch July 6, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Feature New feature sota-implementations/

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vmoens