Summary
Follow-up from review of #2280 (two threads from @leofang). Both host-callback review points share a single root cause: on the stream-capture add_callback path in cuda_core/cuda/core/graph/_graph_builder.pyx, the host node is committed by cuLaunchHostFunc before its slot-table owners are attached, and the post-commit steps can throw — leaving a committed node whose C function pointer references Python objects that then go out of scope (dangling pointer → UB at instantiate/launch).
This is low-risk in practice (narrow trigger; the num_deps invariant holds), so it was intentionally deferred from v1.1.0 rather than starting another review+CI cycle on an approved PR.
Details
-
Missing rollback (capture path). The explicit path GN_callback in _graph_node.pyx wraps _attach_host_callback_owners in try/except: cuGraphDestroyNode(new_node). The capture path add_callback runs _capture_tail_node + _attach_host_callback_owners bare after cuLaunchHostFunc. graph_set_slot is except+ (can throw std::bad_alloc) and returns CUDA_ERROR_NOT_SUPPORTED when the driver lacks user-object support, so the attach step is genuinely fallible.
-
num_deps == 1 raise. _capture_tail_node raises RuntimeError if the stream's post-callback dependency set is not exactly one node. The invariant is expected to hold across capture states, but raising here lands in the same dangling state as (1).
Proposed fix
Make the capture path's post-commit work non-throwing rather than mirroring the cuGraphDestroyNode rollback (destroying a node mid-capture is not clearly legal):
- Call
ensure_slot_table (and the user-object support check) before cuLaunchHostFunc, so the only remaining post-commit work is a map insert.
- Alternatively, attach the callback owners at graph scope for the capture path — the slot table is torn down wholesale at graph destruction, so per-node keying isn't needed here, which also removes
_capture_tail_node and the num_deps question entirely.
- Reassess whether the hard
raise on num_deps != 1 is needed once the ownership is made failure-safe.
Acceptance
- No code path can leave a committed host node in a captured graph with its owners dropped.
- Regression coverage for the capture
add_callback path (owner retained for the graph's lifetime; behavior under a forced attach failure).
Summary
Follow-up from review of #2280 (two threads from @leofang). Both host-callback review points share a single root cause: on the stream-capture
add_callbackpath incuda_core/cuda/core/graph/_graph_builder.pyx, the host node is committed bycuLaunchHostFuncbefore its slot-table owners are attached, and the post-commit steps can throw — leaving a committed node whose C function pointer references Python objects that then go out of scope (dangling pointer → UB at instantiate/launch).This is low-risk in practice (narrow trigger; the
num_depsinvariant holds), so it was intentionally deferred from v1.1.0 rather than starting another review+CI cycle on an approved PR.Details
Missing rollback (capture path). The explicit path
GN_callbackin_graph_node.pyxwraps_attach_host_callback_ownersintry/except: cuGraphDestroyNode(new_node). The capture pathadd_callbackruns_capture_tail_node+_attach_host_callback_ownersbare aftercuLaunchHostFunc.graph_set_slotisexcept+(can throwstd::bad_alloc) and returnsCUDA_ERROR_NOT_SUPPORTEDwhen the driver lacks user-object support, so the attach step is genuinely fallible.num_deps == 1raise._capture_tail_noderaisesRuntimeErrorif the stream's post-callback dependency set is not exactly one node. The invariant is expected to hold across capture states, but raising here lands in the same dangling state as (1).Proposed fix
Make the capture path's post-commit work non-throwing rather than mirroring the
cuGraphDestroyNoderollback (destroying a node mid-capture is not clearly legal):ensure_slot_table(and the user-object support check) beforecuLaunchHostFunc, so the only remaining post-commit work is a map insert._capture_tail_nodeand thenum_depsquestion entirely.raiseonnum_deps != 1is needed once the ownership is made failure-safe.Acceptance
add_callbackpath (owner retained for the graph's lifetime; behavior under a forced attach failure).