|
1 | 1 | # PufferDrive 2.0: A fast and friendly driving simulator for training and evaluating RL agents |
2 | 2 |
|
3 | | -**Daphne Cornelisse¹·*, Spencer Cheng²·*, Pragnay Mandavilli¹, Julian Hunt¹, Kevin Joseph¹, Waël Doulazmi³, Eugene Vinitsky¹** |
| 3 | +**Daphne Cornelisse**¹·$*$, **Spencer Cheng**²·$*$, Pragnay Mandavilli¹, Julian Hunt¹, Kevin Joseph¹, Waël Doulazmi³, Eugene Vinitsky¹ |
4 | 4 |
|
5 | | -¹Emerge Lab at NYU | ²[Puffer.ai](https://puffer.ai/) | ³Valeo | *Shared first authorship |
| 5 | +¹ Emerge Lab at NYU | ² [Puffer.ai](https://puffer.ai/) | ³ Valeo | $*$ Shared first contributor |
6 | 6 |
|
7 | | -*December 12, 2025* |
| 7 | +*December 26, 2025* |
8 | 8 |
|
9 | 9 | --- |
10 | 10 |
|
11 | | -We introduce **PufferDrive 2.0**, a fast and easy-to-use driving simulator for reinforcement learning. It supports training at up to **300,000 steps per second** on a single GPU, enabling agents to reach strong performance in just a few hours. Evaluation and visualization run directly in the browser. |
| 11 | +We introduce **PufferDrive 2.0**, a fast, easy-to-use driving simulator for reinforcement learning (RL). Built on [PufferLib](https://puffer.ai/), it allows you to train agents at **300,000 steps per second** on a single GPU. You can solve thousands of multi-agent scenarios in just 15 minutes. Evaluation and visualization run directly in the browser. |
12 | 12 |
|
13 | | -This post outlines the design goals, highlights the main features, and shows what works out of the box. We conclude with a brief roadmap. |
| 13 | +This post highlights the main features and traces the sequence of projects that led to PufferDrive 2.0. |
14 | 14 |
|
15 | 15 | --- |
16 | 16 |
|
| 17 | +## Highlights |
| 18 | + |
| 19 | +- **Super-fast self-play RL**: Train agents on 10,000 multi-agent Waymo scenarios and reach a near-perfect score in under an hour. [Earlier results](https://arxiv.org/abs/2502.14706) took 24 hours. PufferDrive achieves similar performance in about **15 minutes on a single GPU**. |
| 20 | +- **Long-horizon driving:** Train agents to reach goals indefinitely on large CARLA maps. Demo agents are trained this way. Drive alongside them in the browser below. |
| 21 | +- **Built-in evaluation:** Integrated eval support for the [Waymo Open Sim Agent Challenge (WOSAC)](https://emerge-lab.github.io/PufferDrive/wosac/) and a [human compatibility benchmark](https://emerge-lab.github.io/PufferDrive/evaluation/#human-compatibility-benchmark). |
| 22 | +- **Easy scenario creation:** Edit or design custom scenarios in minutes, including long-tail and stress-test cases, using the [interactive scenario editor](https://emerge-lab.github.io/PufferDrive/scene-editor/). |
| 23 | +- **And more:** Browse the docs for details. |
| 24 | + |
| 25 | + |
| 26 | +## Drive together with trained agents |
| 27 | + |
| 28 | +<iframe src="/assets/game.html" title="PufferDrive Demo" width="1280" height="720" style="border: none; display: block; margin: 2rem auto;"></iframe> |
| 29 | + |
| 30 | +<p style="text-align: center; color: #888; margin-top: 1rem;"> |
| 31 | + Hold <strong>Left Shift</strong> and use arrow keys or <strong>WASD</strong> to control the vehicle. Hold <strong>space</strong> for first-person view and <strong>ctrl</strong> to see what your agent is seeing. |
| 32 | +</p> |
| 33 | + |
| 34 | + |
17 | 35 | ## Introduction and history |
18 | 36 |
|
19 | | -Deep reinforcement learning algorithms, such as [PPO](https://arxiv.org/abs/1707.06347), are highly effective in the billion-sample regime. A consistent finding across domains is that, given sufficient environmental signal and enough data, any precisely specified objective can, in principle, be optimized. |
| 37 | +Deep reinforcement learning algorithms such as [PPO](https://arxiv.org/abs/1707.06347) perform extremely well in the billion-sample regime. RL consistently optimizes precise objectives, even under sparse rewards, if occasional successes occur and the scale is sufficient. |
20 | 38 |
|
21 | | -This shifts the primary bottleneck to simulation. The faster we can generate high-quality experience, the more reliably we can apply RL to hard real-world problems, such as autonomous navigation in dynamic, unstructured environments.[^1] |
| 39 | +This shifts the primary bottleneck to simulation. The faster we can generate high-quality experience, the more reliably we can apply RL to hard real-world problems, such as autonomous navigation in dynamic, multi-agent environments.[^1] |
22 | 40 |
|
23 | | -Over the past few years, several simulators have demonstrated that large-scale self-play can work for driving. Below, we summarize this progression and explain how it led to PufferDrive 2.0. |
| 41 | +Over the past few years, we built several data-driven, multi-agent simulators to study large-scale self-play for driving. We focus on this sequence of projects to show how we arrived at PufferDrive 2.0. |
24 | 42 |
|
25 | | -[^1]: A useful parallel comes from the early days of computing. In the 1970s and 1980s, advances in semiconductor manufacturing and microprocessor design—such as Intel’s 8080 and 80286 chips—dramatically reduced computation costs and increased speed. This made iterative software development accessible and enabled entirely new ecosystems of applications, ultimately giving rise to the personal computer. Multi-agent RL faces a similar bottleneck today: progress is limited by the cost and speed of experience collection. Fast, affordable simulation with integrated RL algorithms may play a similar catalytic role, enabling solutions that were previously out of reach. |
| 43 | +[^1]: A useful parallel comes from the early days of computing. In the 1970s and 1980s, advances in semiconductor manufacturing and microprocessor design—such as Intel’s 8080 and 80286 chips—dramatically reduced computation costs and increased speed. This made iterative software development accessible and enabled entirely new ecosystems of applications, ultimately giving rise to the personal computer. Multi-agent RL faces a similar bottleneck today: progress is limited by the cost and speed of experience collection. Fast, affordable simulation with integrated RL algorithms may play a similar role, enabling solutions that were previously out of reach. |
26 | 44 |
|
27 | 45 | ## Early results with self-play RL in autonomous driving |
28 | 46 |
|
29 | | -[**Nocturne**](https://arxiv.org/abs/2206.09889) was the first paper to show that self-play RL could work for driving at scale. Using maps from the [Waymo Open Motion Dataset (WOMD)](https://waymo.com/open/), PPO agents achieved around an 80% goal-reaching rate without any human data. |
| 47 | +[**Nocturne**](https://arxiv.org/abs/2206.09889) showed that self-play RL could be promising for driving if we have access to a data-driven (grounded) simulator. Using maps from the [Waymo Open Motion Dataset (WOMD)](https://waymo.com/open/), PPO agents trained from scratch in simulation achieved an 80% goal-reaching rate. |
30 | 48 |
|
31 | | -The main limitation was speed. Nocturne ran at roughly 2,000 steps per second, leading to multi-day training times and a complex setup process. |
| 49 | +The main limitation was the _cost_ of simulated experience. Nocturne ran at roughly 2,000 steps per second, so reaching this level of performance required about two days of training on a single GPU. It hinted that self-play RL could work, but generating the required experience was still expensive. |
32 | 50 |
|
33 | | -The results were promising, but it was clear that scale was a major constraint. |
| 51 | +This suggested that the primary bottleneck was simulation throughput, not the learning algorithm. |
34 | 52 |
|
35 | 53 | ## Scaling up |
36 | 54 |
|
37 | | -Subsequent work showed what becomes possible when scale is no longer the bottleneck. |
| 55 | +Later work explored what becomes possible once reaching scale is no longer a bottleneck. |
| 56 | + |
| 57 | +* [**Gigaflow**](https://arxiv.org/abs/2501.00678) demonstrated that large-scale self-play alone can produce robust, naturalistic driving. With a batched simulator, it trained on the equivalent of decades of driving per hour and achieved strong performance across multiple benchmarks without human driving demonstrations. |
| 58 | +* [**GPUDrive**](https://arxiv.org/abs/2408.01584), built on [Madrona](https://madrona-engine.github.io/), achieved similar controllability in about one day on a single consumer GPU using a simpler reward function and standard PPO. |
38 | 59 |
|
39 | | -* [**Gigaflow**](https://arxiv.org/abs/2501.00678) demonstrated that large-scale self-play alone can produce robust, naturalistic driving. Using a highly batched simulator, it trained on the equivalent of **decades of driving experience per hour** and achieved state-of-the-art performance across multiple autonomous driving benchmarks—without using any human data. |
40 | | -* [**GPUDrive**](https://arxiv.org/abs/2408.01584), built on [Madrona](https://madrona-engine.github.io/), showed that [similar behavior could be learned](https://arxiv.org/abs/2502.14706) in about one day on a single consumer GPU, using a simple reward function and a standard PPO implementation. |
| 60 | +These results suggested that once simulation becomes cheap, self-play RL can produce robust autonomous driving policies without relying on human demonstrations. |
41 | 61 |
|
42 | | -These empirical results support the hypothesis that robust autonomous driving policies can be trained in the billion-sample regime _without any human data_. |
| 62 | + |
| 63 | +**Figure 1:** _Progression of RL-based driving simulators. Left: end-to-end training throughput on an NVIDIA RTX 4080, counting only transitions collected by learning policy agents. Right: wall-clock time to reach 80 percent goal-reaching. This captures both simulation speed and algorithmic efficiency._ |
43 | 64 |
|
| 65 | +| Simulator | End-to-end training SPS | Time to 80% success rate | |
| 66 | +|-----------|--------------|------------------------| |
| 67 | +| Nocturne | 2,000 | ~48 hours | |
| 68 | +| GPUDrive | 50,000 | ~1.7 hours | |
| 69 | +| PufferDrive | 320,000 | ~4 minutes | |
44 | 70 |
|
45 | | - |
46 | | -**Figure 1:** *Progression of RL-based driving simulators. Left: end-to-end training throughput on an NVIDIA RTX 4080, counting only transitions collected by learning policy agents (excluding padding agents). Right: wall-clock time (log scale) required to reach an 80% goal-reaching rate. This metric captures both simulation speed and algorithmic efficiency.* |
47 | 71 |
|
48 | 72 | ## From GPUDrive to PufferDrive |
49 | 73 |
|
50 | | -While GPUDrive delivered impressive raw simulation speed, end-to-end training throughput of around 50K steps per second remained a limiting factor. This was particularly true on large maps such as [CARLA](https://carla.org/). Memory layout and batching overheads, rather than simulation fidelity, became the dominant constraints. |
| 74 | +GPUDrive delivered high raw simulation speed, but end-to-end training throughput (~50K steps/sec) still limited experiments, especially on large maps like [CARLA](https://carla.org/). Memory layout and batching overheads prevented further speedups. |
51 | 75 |
|
52 | | -Faster end-to-end training is critical because it enables tighter debugging loops, broader experimentation, and faster scientific and engineering progress. This led directly to the development of **PufferDrive**. |
| 76 | +We were motivated to get faster end-to-end training because waiting a full day for experimental results slows down everything, debugging, testing, and scientific progress. This led to the development of PufferDrive. |
53 | 77 |
|
54 | | -We partnered with Spencer Cheng from [Puffer.ai](https://puffer.ai/) to rebuild the system around the principles of [**PufferLib**](https://arxiv.org/abs/2406.12905). Spencer reimplemented **GPUDrive**. The result was **PufferDrive 1.0**, reaching approximately 200,000 steps per second on a single GPU and scaling linearly across multiple GPUs. Training agents to solve 10,000 maps from the Waymo datset took roughly 24 hours with GPUDrive. [With PufferDrive, the same results could now be reproduced in about 2 hours](https://x.com/spenccheng/status/1959665036483350994). |
| 78 | +Partnering with Spencer Cheng from [Puffer.ai](https://puffer.ai/), we rebuilt GPUDrive around [**PufferLib**](https://arxiv.org/abs/2406.12905). The result, **PufferDrive 1.0**, reached ~200,000 steps per second on a single GPU and scaled linearly across multiple GPUs. Training agents on 10,000 Waymo maps took roughly 24 hours with GPUDrive—[with PufferDrive, we now reproduce the same results in ~2 hours](https://x.com/spenccheng/status/1959665036483350994). |
55 | 79 |
|
56 | | -## PufferDrive 2.0 |
| 80 | +## Roadmap: PufferDrive 3.0 |
57 | 81 |
|
58 | | -PufferDrive 2.0 builds on this foundation and [TODO]: |
| 82 | +What is next? PufferDrive 3.0 will improve agent diversity, realism, and expand simulation capabilities. Priorities may shift as we test features and gather feedback. You can find an overview of our planned features on the [project board](https://github.com/orgs/Emerge-Lab/projects/7) or open an issue with something you would like to see! |
59 | 83 |
|
60 | | -* Built-in evaluations, including a standard benchmark |
61 | | -* Support for multiple real-world datasets (WOMD, Carla) |
62 | | -* Speed improvement (200K -> 300K) |
63 | | -* Extended browser-based visualization and analysis tools |
| 84 | +**Simulation and environment** |
64 | 85 |
|
65 | | -To our knowledge, PufferDrive 2.0 is among the fastest open-source driving simulators available today, while remaining accessible to new users. |
| 86 | +- 2.5D simulation (allow for maps with overpasses) |
66 | 87 |
|
67 | | -## Highlights |
68 | | -TODO |
| 88 | +**Agent and interaction** |
69 | 89 |
|
70 | | -## Roadmap |
71 | | -TODO |
| 90 | +- Collision checking via grid map |
| 91 | +- Enable initialization of variable agent numbers |
| 92 | +- Fix `active_agent_count` edge cases |
| 93 | +- Support for reward conditioning |
| 94 | +- Lane-based rewards |
| 95 | + |
| 96 | +**Benchmarks** |
| 97 | + |
| 98 | +- More extensive planning benchmark with human replays (more metrics) |
72 | 99 |
|
73 | 100 |
|
74 | 101 | ## Citation |
75 | 102 |
|
76 | | -If you use PufferDrive in your research, please cite: |
| 103 | +If you use PufferDrive, please cite: |
77 | 104 | ```bibtex |
78 | | -@software{pufferdrive2024github, |
| 105 | +@software{pufferdrive2025github, |
79 | 106 | author = {Daphne Cornelisse* and Spencer Cheng* and Pragnay Mandavilli and Julian Hunt and Kevin Joseph and Waël Doulazmi and Eugene Vinitsky}, |
80 | 107 | title = {{PufferDrive}: A Fast and Friendly Driving Simulator for Training and Evaluating {RL} Agents}, |
81 | 108 | url = {https://github.com/Emerge-Lab/PufferDrive}, |
82 | 109 | version = {2.0.0}, |
83 | 110 | year = {2025}, |
84 | 111 | } |
85 | 112 | ``` |
| 113 | + |
86 | 114 | *\*Equal contribution* |
0 commit comments