Performance of llama.cpp on AMD ROCm (HIP) #15021
Replies: 62 comments 114 replies
-
RX 7800 XT (Sapphire Pulse 280W)ggml_cuda_init: found 1 ROCm devices:
build: 00131d6 (6031) ggml_vulkan: Found 1 Vulkan devices:
build: baad948 (6056) Notes:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
build: e725a1a (6104) |
Beta Was this translation helpful? Give feedback.
-
|
Happy to replicate: ggml_cuda_init: found 1 ROCm devices:
build: 9c35706 (6060) On Linux |
Beta Was this translation helpful? Give feedback.
-
RX 7600 XTggml_cuda_init: found 1 ROCm devices:
build: 9c35706 (6060) Running on Linux 6.12.32, mainline amdgpu, ROCm 6.4.1. ggml_vulkan: Found 1 Vulkan devices:
build: 9c35706 (6060) |
Beta Was this translation helpful? Give feedback.
-
|
@olegshulyakov , the 7600 XT actually has a 128 bit memory bus. |
Beta Was this translation helpful? Give feedback.
-
|
AMD MI60. Happy to contribute.
I will post FA=1 and vulkan results once I have time during the weekend. |
Beta Was this translation helpful? Give feedback.
-
MI100Using ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -sm none -mg 0
build: 9c35706 (6060) I'm running Ubuntu 24.04.2 and ROCm 6.4.1 |
Beta Was this translation helpful? Give feedback.
-
|
I expected it to be faster than RX 7800 XT because of HBM2... Have you tried to launch with a single device only? |
Beta Was this translation helpful? Give feedback.
-
|
Bandwidth utilization is still fairly low on the gcn/cdna parts (gcn/cdna = same thing for tg). |
Beta Was this translation helpful? Give feedback.
-
AMD Instinct MI300Xroot@0-4-9-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 2bf3fbf (6069) Ref: #14640 |
Beta Was this translation helpful? Give feedback.
-
|
I'm just referring to the rocWMMA flag from the build instructions: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip
It should work for CDNA too but we have only tested with our RDNA3 cards (7900 XTX) and saw huge performance jumps in PP with FA on: #10879 (reply in thread) Please try it out because 1/3rd the performance in PP with FA on is just... strange at best |
Beta Was this translation helpful? Give feedback.
-
|
With root@0-4-9-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: ee3a9fc (6090) |
Beta Was this translation helpful? Give feedback.
-
|
Performance is in the middle between RTX 4090 and RTX 5090. |
Beta Was this translation helpful? Give feedback.
-
|
So rocWMMA does work for CDNA in FA :) |
Beta Was this translation helpful? Give feedback.
-
|
not very well |
Beta Was this translation helpful? Give feedback.
-
Pro V620Why does FA slow down the V620 so much? Been a question I've been trying to answer for a while now.
build: 03d4698 (6074) Linux, ROCm 6.4.1 ( will try upgrading soon) |
Beta Was this translation helpful? Give feedback.
-
|
@samteezy Can you please run per each device PRO V620/Pro WX 3200 and ROCm only backend? |
Beta Was this translation helpful? Give feedback.
-
|
@olegshulyakov The numbers come out the same. Forcing root@llama:~# /root/llama-builds/llama.cpp/bin/llama-bench -m /mnt/models/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -mg 0 -sm none
build: 5c0eb5e (6075) |
Beta Was this translation helpful? Give feedback.
-
Powercolor Hellhound RX 7900 XTX (400W power limit)Opensuse tumbleweed system with rocm packages from
build: 5c0eb5e (6075) Sapphire Nitro 7900 XTX (400W power limit)In a different PC unfortunately because these GPUs are too chonky to fit in a regular case
build: 9c35706 (6060) |
Beta Was this translation helpful? Give feedback.
-
|
So what are recommend settings/what to do to get best performance on 7900 XTX ? I have SAPPHIRE NITRO+ AMD Radeon RX 7900 XTX Vapor-X 24GB and without changing anything I get WAY worse results.
build: b9be58d (1005) And this is what LACT showed while running it 
|
Beta Was this translation helpful? Give feedback.
-
|
At these high speeds with fast gpus the cpu gets important for results, your results and his would be within the expected variance for differing cpu performance. Not that it really matters mutch as the cpu gets almost irrelevant when a model of a size to fill the device is used. Benchmarking llama 7B Q4_0 is not really that great, as it dosent reflect actual usage much, this hurts the most on cdna devices which scale better than you would expect performance wise when increasing the number parameters. |
Beta Was this translation helpful? Give feedback.
-
|
Go for ROCm 7.0, it has officially been released now. And compile llama.cpp with ROCWMMA enabled, see #15021 (comment) . You should get much better results. |
Beta Was this translation helpful? Give feedback.
-
|
ROCm 7 is released? that's great news! I'll try it out as soon at is lands in my distro's package manager. That said I don't think it'll give any performance improvements for the 7900 XTX. The reason 9070 XT gets a boost with rocm7 is because pre rocm7 WMMA is not implemented for RDNA4. But we'll see. BTW FYI in my benchmarks, the powercolor 7900XTX was paired with a ryzen 2700 with 64GB RAM, and the sapphire 7900XTX was paired with a 5700X3d also 64GB RAM. I did not see an appreciable performance difference between the two setups in LLM inference. |
Beta Was this translation helpful? Give feedback.
-
|
I see. To me that 3573 vs 3053 seems big difference. I'm running this on AMD Threadripper 1920X (12 core) with PCIe 3.0 x16. What benchmark I could use to compare more accurate real world inference performance? Now when I booted with |
Beta Was this translation helpful? Give feedback.
-
Powercolor Red Devil 7900XTXAdrenalin 25.8.1 just came out, so time to test again
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
build: 2572689 (6099) Still lower than the historical highs on May 26th (3599 and 3743), and a loss and a win against July 22nd (3529 and 3598). |
Beta Was this translation helpful? Give feedback.
-
|
RX 7900 XTX (ASUS TUF)
build: 6c7e9a5 (6118) |
Beta Was this translation helpful? Give feedback.
-
RX 6800 (16GB 203W)ROCm 6.3.4 on Ubuntu 24.04 in a Docker container
build: 79c1160 (6123) Bonus benchmarksI ran these to compare ROCm versions on various models. Obviously the results are specific to my RX 6800 and shouldn't be used to make any judgments about ROCm performance in general, especially on RDNA3 and later gpus. I use 6.3.4 because I don't care about LLama 3 8B. Note how fast the new MoE models are - gpt-oss-20B even at Q6_K_XL is faster than this 7B Q4_0 model. (Do make sure that you have a fixed version because the original gpt-oss releases had some issues - I used https://huggingface.co/unsloth/gpt-oss-20b-GGUF). ROCm 6.3.4
ROCm 6.4.3
|
Beta Was this translation helpful? Give feedback.
-
RX 6800 (16GB 203W)ROCm 7.2.4 arch 7.0.10-arch1-1 |
Beta Was this translation helpful? Give feedback.
-
|
RX 7900 XTX (ASUS TUF a bit overclocked for 100 mhz for core and VRAM) ./build/bin/llama-bench -m /home/vk/Downloads/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 648ebcd (6146) |
Beta Was this translation helpful? Give feedback.
-
|
RX 6900 XT AMD Reference Card (Stock clocks) Debian Testing llama.cpp version: gguf-v0.17.1-386-gfd1234cb
|
Beta Was this translation helpful? Give feedback.
-
|
@tdjb Results are pretty low, can you re-test using llama.cpp standalone without Docker? |
Beta Was this translation helpful? Give feedback.
-
|
Just a quick test, installed the llama.cpp build from Debian sid (was surprised to even find a build to be honest), which appears to be b5882 and the results came in quite similar. I tried the benchmark on both of my devices, as one is on a slower PCIe 4x slot, the results below are from the faster run. Why do you think the 6900 XT should perform better?
Happy to run further tests. |
Beta Was this translation helpful? Give feedback.
-
|
It should be 10% better on my understanding, according to specs: RX 6900 XT and RX 6800 XT |
Beta Was this translation helpful? Give feedback.
-
|
the rdna2 results are mostly surprisingly high, given the hardware capabilities, not low. |
Beta Was this translation helpful? Give feedback.
-
|
GigaByte R9700
|
Beta Was this translation helpful? Give feedback.
-
|
So 7900 XTX has better performance, that's sad. Also weird that Vulkan performs way worse on pp than ROCm but better on tg. |
Beta Was this translation helpful? Give feedback.
-
|
We'll see when someone a bit more experienced gives it a shot. My benchmarks are about as vanilla as it gets. Threw it in an unraid server (12700k and 128GB DDR4-2133), made docker images and ran benchmarks. Many of the 7900 XTX results are baremetal, have factory overclock or are manually overclocked, installed additional drivers, and/or have raised power limits. I bet someone will beat my benchmarks shortly. |
Beta Was this translation helpful? Give feedback.
-
there is zero reason to expect 9070(xt) to perform better than the xtx |
Beta Was this translation helpful? Give feedback.
-
Radeon RX 9070 (non-XT)
build: 65349f2 (6183) I tried to enable the use of rocwmma with Still surprising that these numbers are better than the 9070 XT. |
Beta Was this translation helpful? Give feedback.
-
|
Phew went through some issues here. ASRock Radeon AI PRO R9700. Rocm 7.2Stay away from rocm7.2: Rocm 7.1.1 |
Beta Was this translation helpful? Give feedback.
-
|
that worked! getting similar performance with rocm 7.2 now |
Beta Was this translation helpful? Give feedback.
-
|
I'm seeing that the --amdgpu-unroll-threshold-local=600 flag is being ignored at compile time when llama.cpp is building the hip.so artifacts. Is this a more recent change and is the regression back with rocm7.2? |
Beta Was this translation helpful? Give feedback.
-
|
@pedapudi You have to set it in |
Beta Was this translation helpful? Give feedback.
-
|
*on linux, on windows you must set DCMAKE_CXX_FLAGS instead |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I set it using -DCMAKE_HIP_FLAGS on Ubuntu. |
Beta Was this translation helpful? Give feedback.
-
|
Running on Manjaro Linux, using the ROCm docker image:
build: 1a29907 (8202) |
Beta Was this translation helpful? Give feedback.
-
AMD Radeon RX 7900 XTX (Sapphire Pulse 7900 XTX)System details
ROCm results
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 24560 MiB):
build: 8ff0207 (8400) Vulkan results
ggml_vulkan: Found 1 Vulkan devices:
build: bcdc4eb (8400) |
Beta Was this translation helpful? Give feedback.
-
|
No ROCMinfo on Windows, but: AMD Radeon RX 7900 XTX reference card, watercooled: ROCM 7.2:
build: 1e64534 (8429) Vulkan:
build: c9ced49 (7549) Newer Vulkan version gives me the same speed as ROCM |
Beta Was this translation helpful? Give feedback.
-
|
what vulkan veresion? |
Beta Was this translation helpful? Give feedback.
-
|
I somehow used Vulkan and ROCM together, guess that's a huge speed up. Yeah, that is insane. |
Beta Was this translation helpful? Give feedback.
-
|
:O m8 write a guide on it or something |
Beta Was this translation helpful? Give feedback.
-
|
That is so funny. I know how to do it. I guess I will write a guide tomorrow. PS C:\Users<username>\Downloads\llama-b8429-bin-win-hip-radeon-x64> llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: 07ba6d2 (8417) PS C:\Users<username>\Downloads\llama-b8429-bin-win-hip-radeon-x64> llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: c9ced49 (7549) I had an older Vulkan version installed via winget (build: c9ced49 (7549) - so this is on PATH) and when i use llama-bench in the newest llama HIP version folder it combines them and doubles everything in terms of speed, but that should not work. But it doesl With a newer Vulkan (build: 07ba6d2 (8417))version it doesn't work. I guess it loads the model 2 times and blasts both? |
Beta Was this translation helpful? Give feedback.
-
|
@daMustermann what kind of performance do you get for Qwen3.5-35b-a3b on this card? I have an extremely bad 250 t/s for pp! And I use ROCm 7.1.1. |
Beta Was this translation helpful? Give feedback.
-
|
Vulkan Instance Version: 1.4.341 Linux fedora 6.19.6-200.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC |
Beta Was this translation helpful? Give feedback.
-
|
This thread is for HIP results not vulkan. |
Beta Was this translation helpful? Give feedback.
-
|
@jschoch HINT: try disabling CPU and you will probably see a decrease in pp and an increase in tg |
Beta Was this translation helpful? Give feedback.
-
|
./llama-bench -m ~/models/llama-2-7b.Q4_0.gguf -fa 1,0
build: 406f4e3 (8514) Results above are from using the pre-built binaries, using aur's llama.cpp-hip script gives higher tg128 but slower pp512 llama-bench -m ~/models/llama-2-7b.Q4_0.gguf -fa 1,0
build: unknown (8514) |
Beta Was this translation helpful? Give feedback.
-
|
Adrenalin 26.3.1 WSL Debian 13.3 ROCDXG ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
build: d43375f (8611) |
Beta Was this translation helpful? Give feedback.
-
|
HP g1a laptop with APU capped at 70W. AMD RYZEN AI MAX+ PRO 395 with 128GB ram using TheRock 7.12 $ llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0 -p 512,1024,2048,4096,8192,16384,32768 -n 128,256,512,1024
$ llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 1 -p 512,1024,2048,4096 -n 128,256
$ llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 1 -p 8192 -n 1024
|
Beta Was this translation helpful? Give feedback.
-
|
llama-bench -m ~/models/llama-2-7b.Q4_0.gguf -fa 1,0
build: e365e65 (8851) |
Beta Was this translation helpful? Give feedback.
-
|
llama-bench -m ~/models/llama-2-7b.Q4_0.gguf -fa 1,0
build: 5757c4d (9090) |
Beta Was this translation helpful? Give feedback.
-
|
ROCm 7.2.3 /workspace/llama.cpp/build/bin/llama-bench -m /data2/llm/models/llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1 -p 512 -n 128 --device ROCm0
build: 046e284 (9085) 2 card on PCI 4
build: 046e284 (9085) |
Beta Was this translation helpful? Give feedback.
-
|
Is the "AMD Radeon Graphics, gfx1201 (0x1201) -> AI PRO R9700 (ASRock)-> PCI 4x4" by any chance on chipset PCIe or via M.2 ? I found Chipset PCIe VERY unstable and rocm-bandwidth-test -A (or -a) very inconsistent and wildly wayyyy below it's theoretical speed. I've ended up using bifurcation on my B650M and running PCIe 4.0 x8/x8 (wigh iGPU disabled as it was also messing the bandwidth somehow). |
Beta Was this translation helpful? Give feedback.
-
|
rocm Version : 7.13.0a20260508-1
|
Beta Was this translation helpful? Give feedback.
-
Radeon 890M
Running the more detailed test of: Getting results of:
build: 338ea1e (9127) |
Beta Was this translation helpful? Give feedback.
-
|
A data point from the image-diffusion side, since the rocWMMA flash-attn numbers here are (understandably) all LLM token-generation. Diffusion attention is a different shape — non-GQA ( Setup: AMD Radeon AI PRO R9700 (Navi 48 / RDNA4 / gfx1201, 32 GB),
So ~6% on the attention kernel, ~2.5% on total wall, output verified correct at the same seed. Two things worth noting for other AMD/diffusion users:
The small gain is consistent with this workload being memory-bandwidth-bound at 1024² (O(n²) over 4096 tokens), not matmul-FLOP-bound — matrix cores help, just not a lot here. The same card sees a bigger relative benefit at smaller, more compute-bound configs. Happy to run more diffusion points (resolutions/steps/other RDNA cards) if it's useful to the rocWMMA tuning work in #16827 — it's a fixed-context, non-GQA stress case the LLM benchmarks don't cover. |
Beta Was this translation helpful? Give feedback.
-
|
it should not displatch the tile kernel for the non rocwmma case for most shapes, it should dispatch the mma kernel: llama.cpp/ggml/src/ggml-cuda/fattn.cu Line 519 in 354ebac for CDNA this is the fastest kernel and for RDNA4 this kernel should also be fastest, at least for the shapes used in llms. |
Beta Was this translation helpful? Give feedback.
-
|
Quick follow-up to my earlier diffusion numbers — the rocWMMA flash-attn win on RDNA4 turns out to be strongly config-dependent, which lines up nicely with the LLM results others have posted in this thread. Same setup as before (R9700 / gfx1201 / ROCm 7.0.2 / rocWMMA 2.0, FLUX.1 Krea Q4,
So at small / compute-bound configs the matrix-core path is a big win (~2.4×), but as resolution grows the attention becomes memory-bandwidth-bound (O(n²) K/V streaming) and the gain shrinks to a few percent. Same shape as the LLM side here — large on long-context prefill, modest on short prompts / decode. Net for image-diffusion users on RDNA4: definitely worth building with rocWMMA fattn on (ROCm 7 / rocWMMA 2.0) — just be aware the upside is very workload-dependent. For reference, with FA on the R9700 lands right next to an A6000 (CUDA) at the same diffusion recipe. |
Beta Was this translation helpful? Give feedback.
-
|
you should probably not be useing rocwmma fattn at all on gfx12, its slower than the default fattn implementation on this arch at least for llms |
Beta Was this translation helpful? Give feedback.
-
|
I get better results with rocwmma on my R9700 and Gemma 4 31B for prompt processing. |
Beta Was this translation helpful? Give feedback.
-
|
OS: CachyOS
build: ba5b911 (9518) Just for gigs, Vulkan, it is quite a bit faster.
|
Beta Was this translation helpful? Give feedback.
-
|
9070 xt > 7900 xtx ? I'm so surprised... by 162% more PP ;-) |
Beta Was this translation helpful? Give feedback.
-
|
Funnily enough these are the results for my 7900 XT on the same PCIe 4.0 x8 slot, it has quite a bit improved TG from what I see. Vulkan is more than 10% faster than ROCm in TG and just a smidget slower in PP. GGML_VK_VISIBLE_DEVICES=1 llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
And for ROCm results: CUDA_VISIBLE_DEVICES=0 llama-bench -m llama-2-7b.Q4_0.gguf -ngl 99 -fa 0,1
|
Beta Was this translation helpful? Give feedback.
-
In all honesty I prefer TG any day than PP :) I am more surprised how well Vulcan backend performs, AMD should be ashamed. |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Performance of llama.cpp on Apple Silicon M-series, Performance of llama.cpp on Nvidia CUDA and Performance of llama.cpp with Vulkan, but for ROCm! I think it's good to consolidate and discuss our results here.
We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our ROCm(HIP) releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and ROCm info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same device I'll prioritize newer commits with substantial ROCm updates, otherwise I'll pick the one with the highest overall score at my discretion. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!
ROCm Scoreboard for Llama 2 7B, Q4_0 (no FA)
ROCm Scoreboard for Llama 2 7B, Q4_0 (with FA)
More detailed test
The main idea of this test is to show a decrease in performance with increasing size.
Beta Was this translation helpful? Give feedback.
All reactions