Skip to content

cuda.core graph: host-callback capture path can leave a committed, unowned node #2317

Description

@Andy-Jost

Summary

Follow-up from review of #2280 (two threads from @leofang). Both host-callback review points share a single root cause: on the stream-capture add_callback path in cuda_core/cuda/core/graph/_graph_builder.pyx, the host node is committed by cuLaunchHostFunc before its slot-table owners are attached, and the post-commit steps can throw — leaving a committed node whose C function pointer references Python objects that then go out of scope (dangling pointer → UB at instantiate/launch).

This is low-risk in practice (narrow trigger; the num_deps invariant holds), so it was intentionally deferred from v1.1.0 rather than starting another review+CI cycle on an approved PR.

Details

  1. Missing rollback (capture path). The explicit path GN_callback in _graph_node.pyx wraps _attach_host_callback_owners in try/except: cuGraphDestroyNode(new_node). The capture path add_callback runs _capture_tail_node + _attach_host_callback_owners bare after cuLaunchHostFunc. graph_set_slot is except+ (can throw std::bad_alloc) and returns CUDA_ERROR_NOT_SUPPORTED when the driver lacks user-object support, so the attach step is genuinely fallible.

  2. num_deps == 1 raise. _capture_tail_node raises RuntimeError if the stream's post-callback dependency set is not exactly one node. The invariant is expected to hold across capture states, but raising here lands in the same dangling state as (1).

Proposed fix

Make the capture path's post-commit work non-throwing rather than mirroring the cuGraphDestroyNode rollback (destroying a node mid-capture is not clearly legal):

  • Call ensure_slot_table (and the user-object support check) before cuLaunchHostFunc, so the only remaining post-commit work is a map insert.
  • Alternatively, attach the callback owners at graph scope for the capture path — the slot table is torn down wholesale at graph destruction, so per-node keying isn't needed here, which also removes _capture_tail_node and the num_deps question entirely.
  • Reassess whether the hard raise on num_deps != 1 is needed once the ownership is made failure-safe.

Acceptance

  • No code path can leave a committed host node in a captured graph with its owners dropped.
  • Regression coverage for the capture add_callback path (owner retained for the graph's lifetime; behavior under a forced attach failure).

Metadata

Metadata

Assignees

Labels

P1Medium priority - Should dobugSomething isn't workingcuda.coreEverything related to the cuda.core module

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions