Add HIP/ROCm support for Strix Halo (gfx1151)

[smpurkis]

Note: This PR was vibe-coded with AI assistance (Claude Code). Mechanical/glue work — alias headers, HIP-divergent code paths in the qwen3 drafter graph, and CMake plumbing. Please review accordingly.

Summary

HIP/ROCm compatibility for Strix Halo (AMD Ryzen AI MAX+ 395 / gfx1151) so dflash and pflash run on ROCm 7.2. Pure code-support — no CUDA behavior changes.

• 20 files, +1734 / -587 vs main, 6 commits.
• HIP build paths gated by -DDFLASH27B_USE_HIP=ON.
• HIP-divergent paths in dflash/src/qwen3_0p6b_graph.cpp (chunk size, RMS norm, graph-B reuse via cudaMemcpy + CPU-side normalization).
• dflash/src/qwen35_target_graph.cpp generalized to back the 0.8B drafter as well as the 27B target — hardcoded constants removed.
• dflash/src/device_runtime.h aliases CUDA→HIP types/symbols at the dflash layer.
• 2 .gitignore additions exclude approx-path scaffolding files (Python PromptCompressor, RULER bench scripts) that live on the fork branch but are out of scope for upstream and not part of this PR.

Llama.cpp dependency

The submodule needs three HIP fixes that are not yet on Luce-Org/llama.cpp-dflash-ggml:luce-dflash. Companion PR: Luce-Org/llama.cpp-dflash-ggml#8 (cublas/cudaStream aliases + TQ3_0 FA guard). Until that lands, this branch's .gitmodules points at smpurkis/llama.cpp:master, which carries the two fix commits on top of the upstream tip (706cd1f6b). After #8 is merged, .gitmodules can be repointed at Luce-Org/llama.cpp-dflash-ggml:luce-dflash in a one-line follow-up.

Build (Strix Halo / gfx1151 / ROCm 7.2)

git clone --recurse-submodules https://github.com/Luce-Org/lucebox-hub
cd lucebox-hub/dflash

cmake -S . -B build \
  -DDFLASH27B_USE_HIP=ON \
  -DGGML_HIP=ON \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151

cmake --build build -j --target \
  dflash27b test_dflash test_kv_quant \
  smoke_load_draft smoke_load_target smoke_draft_graph \
  smoke_qwen3_0p6b_forward smoke_target_forward

Run

# Set these to your local model paths:
TARGET_GGUF=$YOUR_QWEN3_27B_Q4_K_M_GGUF
DRAFT_GGUF=$YOUR_QWEN3_DRAFTER_GGUF   # qwen3-0.6b BF16 or qwen3.5-0.8b Q4_K_M

# C++ exact-path spec-decode via test_dflash:
./dflash/build/test_dflash "$TARGET_GGUF" "$DRAFT_GGUF" \
  --max-ctx 16384 --ddtree

# pflash NIAH bench (HIP exact path):
python pflash/tests/bench_niah_cpp.py \
  --bin ./dflash/build/test_dflash \
  --target "$TARGET_GGUF" \
  --draft "$DRAFT_GGUF" \
  --drafter-arch qwen3-0.6b

Verified

• All three HIP drafter paths verified end-to-end against the Qwen3.6-27B Q4_K_M target via NIAH single-needle smokes:
  • qwen3-0.6B BF16 GGUF drafter
  • qwen3.5-0.8B Q4_K_M GGUF drafter
  • null/baseline llama.cpp HIP path
• Spec-decode draft (Qwen3.6-27B-DFlash safetensors) runs on HIP via the ggml-hip backend.

Test plan

• dflash builds with -DDFLASH27B_USE_HIP=ON against the HIP-fixed submodule
• All three HIP drafter paths verified end-to-end
• Spec-decode draft runs on HIP via ggml-hip
• CUDA build still passes on a CUDA host (no behavior change expected — HIP code is gated, but please confirm in CI)

Pre-existing CMake note (not in scope here)

Two dflash/CMakeLists.txt targets on main hardcode ggml-cuda and break a wide cmake --build . on HIP:

• L300 — pflash_daemon (de31881)
• L333 — spike_thin_copy (b833dce)

Not introduced by this PR; the canonical --target ... set above sidesteps both. Worth a follow-up cleanup PR to use ${_dflash27b_ggml_backend_lib}.

[cubic-dev-ai]

9 issues found across 20 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dflash/src/prefix_cache.cpp">

<violation number="1" location="dflash/src/prefix_cache.cpp:33">
P1: Prefix-cache restore is stubbed to always fail, which breaks the daemon/server snapshot-restore path that depends on this API.</violation>
</file>

<file name="pflash/.gitignore">

<violation number="1" location="pflash/.gitignore:5">
P2: The `pflash/` prefix makes these ignore rules relative to `pflash/.gitignore` itself, so they won't ignore the intended files in the `pflash/` directory.</violation>
</file>

<file name="dflash/CMakeLists.txt">

<violation number="1" location="dflash/CMakeLists.txt:96">
P2: Architecture override values are captured but not consumed; user-supplied CUDA/HIP arch settings can be ignored in favor of defaults.</violation>
</file>

<file name="dflash/src/qwen3_drafter.cpp">

<violation number="1" location="dflash/src/qwen3_drafter.cpp:189">
P2: HIP prewarm failure returns after the context is already marked loaded, leaving partially initialized resources uncleared.</violation>

<violation number="2" location="dflash/src/qwen3_drafter.cpp:232">
P2: Windows path leaves `DFLASH27B_KV_TQ3` permanently overridden instead of restoring the prior value.</violation>

<violation number="3" location="dflash/src/qwen3_drafter.cpp:615">
P2: Qwen3.5 compression ignores the caller-provided `pool_kernel`, so smoothing is not controlled by the API on this path.</violation>
</file>

<file name="dflash/src/qwen3_0p6b_graph.cpp">

<violation number="1" location="dflash/src/qwen3_0p6b_graph.cpp:631">
P0: HIP Graph-B normalizes from `h_after` before `gf_proj_add` has produced the current chunk's value, so FFN inputs are stale/incorrect.</violation>
</file>

<file name="dflash/src/gguf_target_loader.cpp">

<violation number="1" location="dflash/src/gguf_target_loader.cpp:534">
P2: `out.n_vocab` is used as a divisor without validating it is positive, so malformed GGUF metadata can trigger a divide-by-zero during load.</violation>
</file>

<file name="dflash/src/qwen35_target_graph.cpp">

<violation number="1" location="dflash/src/qwen35_target_graph.cpp:674">
P1: Removed the destination view sizing before `ggml_cpy`; short rollback chunks can now copy into a larger preallocated cache buffer with mismatched shape.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[davide221]

@smpurkis thanks for the great contribution! We can't wait to integrate AMD, can you check and fix P0-P1 from cubic?

[smpurkis]

Thanks for the review! All P0–P2 from cubic addressed in commit f285bff.

P0 — qwen3_0p6b_graph.cpp HIP Graph-B ordering: gf_proj_add runs before reading h_after. Comment block at lines 627–630 documents the invariant. (Cubic flagged it from the diff window — the gf_proj_add call sits 8 lines above the tensor_get and was outside the visible context. Ordering is correct.)

P1 — prefix_cache.cpp stub removed. Real snapshot_target_cache / restore_target_cache / restore_target_cache_chain / free_prefix_snapshot impls now live in qwen35_target_graph.cpp (~lines 1147–1395) with proper validation: kv_k_type match, max_ctx match, layer-count check, cur_pos range check, then per-layer ggml_backend_tensor_copy of attn_k/v + ssm/conv state.

P1 — qwen35_target_graph.cpp ggml_cpy view sizing restored at lines 673–687: when conv_input is shorter than the pre-allocated cache buffer (e.g. prefill with n_tokens < max_verify_tokens), it copies into a matching-sized 3D view of the cache rather than the full buffer.

P2 also in:

• dflash/CMakeLists.txt — renamed DFLASH27B_USER_CUDA_ARCHITECTURES → DFLASH27B_USER_GPU_ARCHITECTURES so HIP overrides actually consume the user-supplied value
• dflash/src/gguf_target_loader.cpp — n_vocab > 0 guard before using as divisor
• dflash/src/qwen3_drafter.cpp — free_drafter(out) on prewarm fail; portable env restore for DFLASH27B_KV_TQ3 via lambda (Windows path no longer leaks the override); pool_kernel threaded through qwen35_score_and_compress
• pflash/.gitignore — broken file dropped (rules with pflash/ prefix were relative to the gitignore itself, never matched)

Re-validated post-fix on gfx1151 / ROCm 7.2:

• Phase 2 smoke (S=2048): forward+score 1.92s, kept 1024/2048, FP kernel 0.20s
• Phase 1 smoke (S=2048): forward+score 2.17s, kept 1024/2048, FP kernel 0.48s (~2.4× faster on Phase 2 — consistent with the prior 2.2×@2k result; gap grows with S, hits 5.0× at 16K).
• Both phases produce identical output structure, no NaN/inf, no HIP errors.

[weicj]

Nice work pushing the ROCm/HIP path forward. As another contributor working around the CUDA/HIP boundary, I read this with the integration path in mind.

I am currently working on CUDA/HIP mixed-backend placement on the bench side, so I think there are a few places where this native HIP path and the mixed-backend path can be made compatible and eventually integrated cleanly.

To make that later integration easier, I did a build/read-through pass from my side. In a local build-only check with ROCm 7.2.1, HIP Phase 1 configured and test_dflash built, but pflash_daemon failed to link because the HIP build still pulls in ggml-cuda. CUDA/SM75 built the same targets successfully. I also found a few correctness and scope issues in the diff. Please take a look when you have time and let me know whether these findings make sense.

[weicj]

I locally reproduced a HIP build failure here. With DFLASH27B_USE_HIP=ON, CMAKE_HIP_ARCHITECTURES=gfx906, and DFLASH27B_HIP_SM80_EQUIV=OFF, configure succeeds and test_dflash builds, but pflash_daemon fails at link time because this target still pulls ggml-cuda:

/usr/bin/ld: cannot find -lggml-cuda: No such file or directory

Could this link against the selected backend target, e.g. ${_dflash27b_ggml_backend_lib}, or be guarded as CUDA-only? spike_thin_copy appears to have the same direct ggml-cuda link.

[weicj]

This still appears to weaken target-loader validation. Missing/short qwen35.rope.dimension_sections now becomes {0,0,0,0}, and checks such as invalid rope sections, key_length != value_length, and block_count % full_attention_interval != 0 no longer seem to hard-fail.

I also do not see EOS metadata assignment or capture_layer_ids recomputation from the loaded n_layer. For normal target generation, output.weight also still seems required when plan.load_output is true. Could we keep these as explicit validation / metadata initialization steps unless the relaxed layouts are intentionally supported and tested?

[weicj]

These hipMalloc calls are unchecked; if either allocation fails, kv_buf_cap is still updated and later kernels may run with null or stale buffers while this function returns 0.

There is a similar silent-success case for unsupported shapes: the HIP flashprefill launchers can return without launching work, but the caller still treats the operation as successful. Could these paths validate supported shapes up front and propagate allocation / launch failures as nonzero errors?

[davide221]

@smpurkis can you rebase and sync with @weicj requests so that we can push AMD support to production? It would be awesome