feat(semaphore): support <cuda/std/semaphore> and <cuda/semaphore> on HIP/AMD#23
feat(semaphore): support <cuda/std/semaphore> and <cuda/semaphore> on HIP/AMD#23jeffdaily wants to merge 2 commits into
Conversation
The counting/binary semaphore implementation is already present and builds on
the cuda::std::atomic wait/notify machinery that is supported on AMD; only the
public umbrella headers were hard-gated with an #error for __HIP_PLATFORM_AMD__.
Remove that gate from both <cuda/std/semaphore> and <cuda/semaphore> so the
existing implementation is exposed. No implementation changes are required; the
remaining sm_70 guard keys on __CUDA_ARCH__ and is inert on AMD.
The semaphore acquire path spins until the count is positive. Threads in a
single wavefront on AMD GPUs do not have independent forward-progress
guarantees, so a blocking acquire() that waits on a release() issued by another
thread of the same wavefront can deadlock; coordinate across separate
wavefronts/blocks (thread_scope_device, thread_scope_system) instead. README.rst
is updated to move the two headers out of the unsupported list and to document
this scope caveat next to the existing device-clock note.
This work was authored with the assistance of an AI coding assistant.
Test Plan:
Validated on a gfx90a GPU (MI250X) with ROCm 7.2.1.
Producer/consumer with cuda::binary_semaphore<thread_scope_device> across two
blocks, and the same pattern with thread_scope_system, both PASS:
hipcc -std=c++17 --offload-arch=gfx90a -I include prod_cons.cpp -o pc && ./pc
Upstream semaphore tests, compiled per .upstream-tests/test/CMakeLists.txt
(HIPCC branch: -include test/force_include_hip.h, which runs each test on host
and launches it on the device), run on gfx90a:
std/thread/thread.semaphore/version.pass.cpp PASS
std/thread/thread.semaphore/max.pass.cpp PASS
std/thread/thread.semaphore/try_acquire.pass.cpp PASS
std/thread/thread.semaphore/acquire.pass.cpp PASS
std/thread/thread.semaphore/release.pass.cpp PASS
heterogeneous/semaphore.pass.cpp PASS
std/thread/thread.semaphore/timed.pass.cpp drives two concurrent agents within
one wavefront through try_acquire_until/try_acquire_for; the timed acquire
reaches its deadline before the same-wavefront releaser runs, which is the
forward-progress limitation described above, so that test is not claimed on
gfx90a.
|
Is this tested via the corresponding semaphore tests? In this PR only the errors are removed. |
Five libcu++ semaphore conformance tests do not exercise the AMD same-wavefront forward-progress hazard, so the `// UNSUPPORTED: hipcc, hiprtc` skip is removed and they run on HIP: version, max, try_acquire (non-blocking), acquire (its two acquires are __syncthreads-separated and each drain a pre-existing count), and the heterogeneous host+device test (acquire and release serialized across separate launches). They pass on AMD CDNA2 (gfx90a) and RDNA3/RDNA4 (gfx1100/gfx1201). release.pass.cpp and timed.pass.cpp keep the skip: both co-schedule a blocking acquire() with its satisfying release() in one wavefront (release completes only because the count starts positive; timed must wait on the sibling and reaches its deadline), and AMD gives no intra-wavefront forward-progress guarantee. Confirmed from the compiled GCN ISA: the safe tests emit no reachable same-wavefront blocking spin, while release/timed emit the s_sleep/s_memrealtime backoff spin co-scheduled with a sibling releaser. Authored with the assistance of Claude (Anthropic).
|
Yes -- and now exercised by the conformance suite directly. I've pushed a commit enabling the semaphore conformance tests that do not hit the AMD same-wavefront forward-progress hazard (removing their Two tests stay
I drew the line from the compiled GCN ISA, not just runtime behavior: the enabled tests have no reachable same-wavefront blocking spin ( |
|
Our general strategy so far is that we do not enable these APIs unless we have full parity. Hence, since we do not support all use cases we would generally keep the API disabled until we can support the full functionality. We could add a flag which can be set to enable these headers in an experimental mode or so. That might be doable if those are needed for any concrete project. Is there any necessity to enable these APIs at the moment? |
Summary
This enables
<cuda/std/semaphore>and the extended<cuda/semaphore>on AMD/HIP by removing the__HIP_PLATFORM_AMD__#errorfrom the two public umbrella headers. The counting and binary semaphore implementation is already present in the tree and builds entirely on thecuda::std::atomicwait/notify machinery that HIP already supports, so this adds and rewrites no semaphore logic; the umbrella#errorwas the only thing making the existing implementation unreachable on AMD.This restores a capability that was available in 2.2.0 (the semaphores compiled before the umbrella gate was introduced) and resolves #10. Code that includes these headers, such as oneflow (
cuda::binary_semaphore<cuda::thread_scope_device>in its embedding LRU cache) and the hipCollections and dgl stacks, pulls the semaphore types through the same umbrellas.What changed
include/cuda/std/semaphore: remove the AMD#errorso the existing__semaphore/*implementation is included.include/cuda/semaphore: remove the AMD#error(this extended header includes<cuda/std/semaphore>, so both must be ungated).README.rst: move semaphore to the supported set, with the forward-progress note below.No implementation files are touched. The remaining
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ < 700guard is inert on AMD, where__CUDA_ARCH__is undefined.Forward-progress note
A blocking
acquire()waits by polling, like the rest of libcu++'s blocking synchronization primitives. AMD GPUs do not provide independent forward progress for threads within a single wavefront, which is the same property that makes libcu++ require sm_70+ Independent Thread Scheduling for these waits on NVIDIA. Consequently a blockingacquire()whose matchingrelease()is issued by another thread of the SAME wavefront is not guaranteed to make progress: whether it completes depends on compiler block ordering and should not be relied upon.Supported patterns:
cuda::thread_scope_deviceorcuda::thread_scope_system, where the producing and consuming wavefronts are scheduled independently. This is the common case (for example oneflow's device-scope usage) and works.try_acquire(), or the timedtry_acquire_for/try_acquire_until, which return instead of spinning so the releasing thread can run.Testing
Built and run on real AMD GPUs:
The
version,max,try_acquire,acquire,release, andheterogeneoussemaphore conformance tests pass on each, and a producer/consumer usingcuda::binary_semaphore<cuda::thread_scope_device>across two blocks acquires and releases correctly, as doescuda::thread_scope_system. The realtime-counter rate used by<cuda/std/chrono>(100 MHz on RDNA) is already defined in the tree and resolves correctly on these targets, confirmed on gfx1201 where the timed paths run within their deadlines.std/thread/thread.semaphore/timed.pass.cpplaunches a producer and a consumer as concurrent agents within a single wavefront, so on AMD its outcome tracks that single-wavefront forward-progress property: the timed acquire can reach its deadline and return false instead of acquiring, depending on how the two agents are scheduled. It is leftUNSUPPORTED: hipccwith a note pointing at the forward-progress section.Test enablement
The six conformance tests above currently carry
UNSUPPORTED: hipcc. They pass on AMD and thehipcctoken can be removed so CI exercises them;timed.pass.cppshould keepUNSUPPORTED: hipccwith a comment referencing the forward-progress note, andhiprtcremains unsupported (not exercised here).Notes
This work was prepared with the assistance of Claude (an AI assistant by Anthropic), which did the analysis, the gate removal, and the validation.