Skip to content

fix Fuzz constant OOM for CUDA#8481

Merged
a10y merged 1 commit into
developfrom
aduffy/debug-cuda-fuzz-oom
Jun 17, 2026
Merged

fix Fuzz constant OOM for CUDA#8481
a10y merged 1 commit into
developfrom
aduffy/debug-cuda-fuzz-oom

Conversation

@a10y

@a10y a10y commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The Fuzz check has been failing for CUDA compress_gpu job for a very long time now (at least a month).

After some investigation, it appears this is because ASAN will mprotect a very large memory region that conflicts with some address space that the CUDA driver wants to own, resulting in cudaCtx initialization failing with OOMs like we've been seeing.

I've run a sample Fuzz workflow with this change and confirmed we are able to get past context init now.

@a10y

a10y commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@codspeed-hq

codspeed-hq Bot commented Jun 17, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 15.58%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 7 improved benchmarks
❌ 1 regressed benchmark
✅ 1573 untouched benchmarks
⏩ 11 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(1000, 10)] 177.7 µs 213.9 µs -16.94%
Simulation take_10k_random 255.8 µs 197.8 µs +29.27%
Simulation take_10k_contiguous 276.3 µs 218.5 µs +26.46%
Simulation patched_take_10k_contiguous_patches 291 µs 232.3 µs +25.26%
Simulation patched_take_10k_random 303 µs 244.2 µs +24.07%
WallTime cuda/bitpacked_u8/unpack/3bw[100M] 352 µs 301.7 µs +16.69%
Simulation bitwise_not_vortex_buffer_mut[128] 215.3 ns 186.1 ns +15.67%
Simulation bitwise_not_vortex_buffer_mut[1024] 275.6 ns 246.4 ns +11.84%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing aduffy/debug-cuda-fuzz-oom (18787a8) with develop (85aad72)

Open in CodSpeed

Footnotes

  1. 11 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

cargo-fuzz builds the compress_gpu target with AddressSanitizer, which
mprotects the shadow gap PROT_NONE. That collides with the large
virtual-address range the CUDA driver reserves, so CudaSession::try_default()
fails and the target aborts on the very first input ("Failed to initialize
CUDA device 0" at vortex-cuda/src/session.rs). Setting protect_shadow_gap=0
stops ASan protecting the gap and frees the VA range for CUDA, letting device
init succeed.

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@a10y a10y force-pushed the aduffy/debug-cuda-fuzz-oom branch from 0d37fb9 to 18787a8 Compare June 17, 2026 20:40
@a10y a10y changed the title (DO NOT MERGE) debug the CUDA Fuzz OOMing fix Fuzz constant OOM for CUDA Jun 17, 2026
@a10y a10y marked this pull request as ready for review June 17, 2026 20:41
@a10y a10y requested a review from a team June 17, 2026 20:41
@0ax1 0ax1 self-requested a review June 17, 2026 20:43
@a10y a10y enabled auto-merge (squash) June 17, 2026 20:43

@0ax1 0ax1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy when CI is happy 🙂

@0ax1 0ax1 added the changelog/fix A bug fix label Jun 17, 2026
@a10y a10y closed this Jun 17, 2026
auto-merge was automatically disabled June 17, 2026 21:33

Pull request was closed

@a10y a10y reopened this Jun 17, 2026
@a10y a10y enabled auto-merge (squash) June 17, 2026 21:51
@a10y a10y merged commit 69ce1ed into develop Jun 17, 2026
139 of 141 checks passed
@a10y a10y deleted the aduffy/debug-cuda-fuzz-oom branch June 17, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants