Skip to content

Commit 3b921a7

Browse files
MagicalTuxclaude
andcommitted
Free the CUDA context on Gpu drop (fix cuCtxCreate OOM)
Gpu had no Drop impl, so every finished fragment leaked its CUDA context and loaded module. Each context reserves a few hundred MB of device memory, so after a few dozen fragments cuCtxCreate started failing with "out of memory" and the worker spun claiming fragments it could never run. Add Drop for Gpu (cuModuleUnload + cuCtxDestroy) and drop the now-stale allow(dead_code) on the ctx field. Add an ignore-gated GPU regression test that creates/drops a Gpu 64 times: it passes with the fix and dies at iteration ~56 with the exact OOM without it. Bump version to 0.1.7. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 64a0be6 commit 3b921a7

3 files changed

Lines changed: 47 additions & 3 deletions

File tree

Cargo.lock

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "decryptd"
3-
version = "0.1.6"
3+
version = "0.1.7"
44
edition = "2024"
55
license = "Proprietary"
66
authors = ["Karpeles Lab Inc"]

src/cuda.rs

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ unsafe extern "C" {
3333
fn cuCtxCreate_v2(pctx: *mut CuContext, flags: u32, dev: CuDevice) -> CuResult;
3434
fn cuCtxDestroy_v2(ctx: CuContext) -> CuResult;
3535
fn cuModuleLoadData(module: *mut CuModule, image: *const c_void) -> CuResult;
36+
fn cuModuleUnload(module: CuModule) -> CuResult;
3637
fn cuModuleGetFunction(
3738
func: *mut CuFunction,
3839
module: CuModule,
@@ -120,7 +121,6 @@ impl Drop for DeviceBuf {
120121

121122
/// An initialized CUDA context with a module loaded.
122123
pub struct Gpu {
123-
#[allow(dead_code)]
124124
ctx: CuContext,
125125
module: CuModule,
126126
dev: CuDevice,
@@ -207,6 +207,20 @@ impl Gpu {
207207
}
208208
}
209209

210+
impl Drop for Gpu {
211+
/// Release the module and context. Without this every finished fragment leaks
212+
/// its CUDA context; after enough fragments `cuCtxCreate` starts failing with
213+
/// `out of memory` (each context reserves device memory) and no further work
214+
/// runs. `cuCtxDestroy` alone frees the module too, but unload it explicitly
215+
/// so the ordering mirrors acquisition.
216+
fn drop(&mut self) {
217+
unsafe {
218+
cuModuleUnload(self.module);
219+
cuCtxDestroy_v2(self.ctx);
220+
}
221+
}
222+
}
223+
210224
/// Run the generic kernel `entry` over `[start, end]` (inclusive), tiling by `tile`
211225
/// items per launch. `data` is the opaque job blob (uploaded once). Returns the raw
212226
/// output records (`out_count * record_size` bytes, concatenated across tiles), and
@@ -307,3 +321,33 @@ impl DeviceBufView {
307321
)
308322
}
309323
}
324+
325+
#[cfg(test)]
326+
mod tests {
327+
use super::*;
328+
329+
/// Regression test for the leaked-context OOM: create and drop a `Gpu` many
330+
/// times and confirm `cuCtxCreate` keeps succeeding. Before `Drop for Gpu`,
331+
/// each iteration leaked its context and this loop died with "out of memory"
332+
/// after a few dozen rounds. Needs a real GPU + a cubin, so it's `#[ignore]`d;
333+
/// run manually with the cubin path in DECRYPTD_TEST_CUBIN:
334+
/// DECRYPTD_TEST_CUBIN=/path/to/x.sm89.cubin cargo test --release -- --ignored gpu_context
335+
#[test]
336+
#[ignore]
337+
fn gpu_context_is_freed_across_runs() {
338+
let Ok(path) = std::env::var("DECRYPTD_TEST_CUBIN") else {
339+
panic!("set DECRYPTD_TEST_CUBIN to a cubin matching this GPU's arch");
340+
};
341+
let bytes = std::fs::read(&path).expect("read cubin");
342+
// Tag arch 0 so the "skip cubins newer than the GPU" filter always keeps
343+
// it; the real cubin must still match this GPU for cuModuleLoadData.
344+
let cubins = vec![(0u32, bytes)];
345+
for i in 0..64 {
346+
let gpu = Gpu::load_first(&cubins)
347+
.unwrap_or_else(|e| panic!("iteration {i}: load_first failed: {e}"));
348+
// Touch it so the context is really used, then drop at end of scope.
349+
let _ = gpu.compute_capability();
350+
drop(gpu);
351+
}
352+
}
353+
}

0 commit comments

Comments
 (0)