Skip to content

feat(semaphore): support <cuda/std/semaphore> and <cuda/semaphore> on HIP/AMD#23

Open
jeffdaily wants to merge 2 commits into
ROCm:amd-developfrom
jeffdaily:enable-semaphore
Open

feat(semaphore): support <cuda/std/semaphore> and <cuda/semaphore> on HIP/AMD#23
jeffdaily wants to merge 2 commits into
ROCm:amd-developfrom
jeffdaily:enable-semaphore

Conversation

@jeffdaily

Copy link
Copy Markdown

Summary

This enables <cuda/std/semaphore> and the extended <cuda/semaphore> on AMD/HIP by removing the __HIP_PLATFORM_AMD__ #error from the two public umbrella headers. The counting and binary semaphore implementation is already present in the tree and builds entirely on the cuda::std::atomic wait/notify machinery that HIP already supports, so this adds and rewrites no semaphore logic; the umbrella #error was the only thing making the existing implementation unreachable on AMD.

This restores a capability that was available in 2.2.0 (the semaphores compiled before the umbrella gate was introduced) and resolves #10. Code that includes these headers, such as oneflow (cuda::binary_semaphore<cuda::thread_scope_device> in its embedding LRU cache) and the hipCollections and dgl stacks, pulls the semaphore types through the same umbrellas.

What changed

  • include/cuda/std/semaphore: remove the AMD #error so the existing __semaphore/* implementation is included.
  • include/cuda/semaphore: remove the AMD #error (this extended header includes <cuda/std/semaphore>, so both must be ungated).
  • README.rst: move semaphore to the supported set, with the forward-progress note below.

No implementation files are touched. The remaining #if defined(__CUDA_ARCH__) && __CUDA_ARCH__ < 700 guard is inert on AMD, where __CUDA_ARCH__ is undefined.

Forward-progress note

A blocking acquire() waits by polling, like the rest of libcu++'s blocking synchronization primitives. AMD GPUs do not provide independent forward progress for threads within a single wavefront, which is the same property that makes libcu++ require sm_70+ Independent Thread Scheduling for these waits on NVIDIA. Consequently a blocking acquire() whose matching release() is issued by another thread of the SAME wavefront is not guaranteed to make progress: whether it completes depends on compiler block ordering and should not be relied upon.

Supported patterns:

  • Coordinate across separate wavefronts or blocks using cuda::thread_scope_device or cuda::thread_scope_system, where the producing and consuming wavefronts are scheduled independently. This is the common case (for example oneflow's device-scope usage) and works.
  • Within a single wavefront, use the non-blocking try_acquire(), or the timed try_acquire_for / try_acquire_until, which return instead of spinning so the releasing thread can run.

Testing

Built and run on real AMD GPUs:

  • gfx90a (CDNA2, wave64), ROCm 7.2.1
  • gfx1100 (RDNA3, wave32), ROCm 7.2.1
  • gfx1201 (RDNA4, wave32, Windows), ROCm 7.14

The version, max, try_acquire, acquire, release, and heterogeneous semaphore conformance tests pass on each, and a producer/consumer using cuda::binary_semaphore<cuda::thread_scope_device> across two blocks acquires and releases correctly, as does cuda::thread_scope_system. The realtime-counter rate used by <cuda/std/chrono> (100 MHz on RDNA) is already defined in the tree and resolves correctly on these targets, confirmed on gfx1201 where the timed paths run within their deadlines.

std/thread/thread.semaphore/timed.pass.cpp launches a producer and a consumer as concurrent agents within a single wavefront, so on AMD its outcome tracks that single-wavefront forward-progress property: the timed acquire can reach its deadline and return false instead of acquiring, depending on how the two agents are scheduled. It is left UNSUPPORTED: hipcc with a note pointing at the forward-progress section.

Test enablement

The six conformance tests above currently carry UNSUPPORTED: hipcc. They pass on AMD and the hipcc token can be removed so CI exercises them; timed.pass.cpp should keep UNSUPPORTED: hipcc with a comment referencing the forward-progress note, and hiprtc remains unsupported (not exercised here).

Notes

This work was prepared with the assistance of Claude (an AI assistant by Anthropic), which did the analysis, the gate removal, and the validation.

The counting/binary semaphore implementation is already present and builds on
the cuda::std::atomic wait/notify machinery that is supported on AMD; only the
public umbrella headers were hard-gated with an #error for __HIP_PLATFORM_AMD__.
Remove that gate from both <cuda/std/semaphore> and <cuda/semaphore> so the
existing implementation is exposed. No implementation changes are required; the
remaining sm_70 guard keys on __CUDA_ARCH__ and is inert on AMD.

The semaphore acquire path spins until the count is positive. Threads in a
single wavefront on AMD GPUs do not have independent forward-progress
guarantees, so a blocking acquire() that waits on a release() issued by another
thread of the same wavefront can deadlock; coordinate across separate
wavefronts/blocks (thread_scope_device, thread_scope_system) instead. README.rst
is updated to move the two headers out of the unsupported list and to document
this scope caveat next to the existing device-clock note.

This work was authored with the assistance of an AI coding assistant.

Test Plan:
  Validated on a gfx90a GPU (MI250X) with ROCm 7.2.1.

  Producer/consumer with cuda::binary_semaphore<thread_scope_device> across two
  blocks, and the same pattern with thread_scope_system, both PASS:

    hipcc -std=c++17 --offload-arch=gfx90a -I include prod_cons.cpp -o pc && ./pc

  Upstream semaphore tests, compiled per .upstream-tests/test/CMakeLists.txt
  (HIPCC branch: -include test/force_include_hip.h, which runs each test on host
  and launches it on the device), run on gfx90a:

    std/thread/thread.semaphore/version.pass.cpp       PASS
    std/thread/thread.semaphore/max.pass.cpp           PASS
    std/thread/thread.semaphore/try_acquire.pass.cpp   PASS
    std/thread/thread.semaphore/acquire.pass.cpp       PASS
    std/thread/thread.semaphore/release.pass.cpp       PASS
    heterogeneous/semaphore.pass.cpp                   PASS

  std/thread/thread.semaphore/timed.pass.cpp drives two concurrent agents within
  one wavefront through try_acquire_until/try_acquire_for; the timed acquire
  reaches its deadline before the same-wavefront releaser runs, which is the
  forward-progress limitation described above, so that test is not claimed on
  gfx90a.
@obersteiner

Copy link
Copy Markdown
Collaborator

Is this tested via the corresponding semaphore tests? In this PR only the errors are removed.
Also could you explain in which project semaphore is needed?

Five libcu++ semaphore conformance tests do not exercise the AMD same-wavefront forward-progress hazard, so the `// UNSUPPORTED: hipcc, hiprtc` skip is removed and they run on HIP: version, max, try_acquire (non-blocking), acquire (its two acquires are __syncthreads-separated and each drain a pre-existing count), and the heterogeneous host+device test (acquire and release serialized across separate launches). They pass on AMD CDNA2 (gfx90a) and RDNA3/RDNA4 (gfx1100/gfx1201).

release.pass.cpp and timed.pass.cpp keep the skip: both co-schedule a blocking acquire() with its satisfying release() in one wavefront (release completes only because the count starts positive; timed must wait on the sibling and reaches its deadline), and AMD gives no intra-wavefront forward-progress guarantee. Confirmed from the compiled GCN ISA: the safe tests emit no reachable same-wavefront blocking spin, while release/timed emit the s_sleep/s_memrealtime backoff spin co-scheduled with a sibling releaser.

Authored with the assistance of Claude (Anthropic).
@jeffdaily

Copy link
Copy Markdown
Author

Yes -- and now exercised by the conformance suite directly. I've pushed a commit enabling the semaphore conformance tests that do not hit the AMD same-wavefront forward-progress hazard (removing their // UNSUPPORTED: hipcc, hiprtc): version, max, try_acquire, acquire, and the heterogeneous host+device test. They pass on AMD CDNA2 (gfx90a) and RDNA3/RDNA4 (gfx1100/gfx1201). The underlying counting_semaphore/binary_semaphore implementation is unchanged; only the two umbrella-header #error gates were removed.

Two tests stay UNSUPPORTED on HIP, because they co-schedule a blocking acquire() with its satisfying release() in the SAME wavefront, which AMD has no forward-progress guarantee for (lanes of a wavefront share a program counter, so a spin-waiting lane can starve the releasing lane):

  • timed.pass.cpp: acquirer and releaser run in one wavefront with the semaphore starting at 0, so the acquire must wait on the sibling release; the timed acquire reaches its deadline and returns false.
  • release.pass.cpp: the same co-scheduled structure; it completes only because the semaphore starts at a positive count, not because the wavefront makes independent progress -- so it would be a deadlock-capable gate.

I drew the line from the compiled GCN ISA, not just runtime behavior: the enabled tests have no reachable same-wavefront blocking spin (acquire's two acquires are __syncthreads-separated and each drain a pre-existing count; try_acquire is non-blocking; version/max are compile-time only; the heterogeneous test serializes acquire and release across separate launches), whereas release/timed emit a reachable s_sleep/s_memrealtime backoff spin co-scheduled with a sibling releaser. The README note documents that same-wavefront blocking acquire()/release() is unsupported on AMD (use cross-wavefront placement, or the non-blocking/timed APIs).

@obersteiner

Copy link
Copy Markdown
Collaborator

Our general strategy so far is that we do not enable these APIs unless we have full parity. Hence, since we do not support all use cases we would generally keep the API disabled until we can support the full functionality. We could add a flag which can be set to enable these headers in an experimental mode or so. That might be doable if those are needed for any concrete project. Is there any necessity to enable these APIs at the moment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA]: semaphore not supported in latest release

2 participants