I ran the same small dense model across five quantization levels on a Raspberry Pi 5 while keeping the runtime, thread count, clocks and measurement method fixed. I wanted to see whether decode on a small ARM system showed the same kind of memory-wall behavior I have worked on before: as the weight footprint gets larger, does token rate fall in roughly the opposite direction?
Setup: Raspberry Pi 5, 4x Cortex-A76, governor performance at 2.4 GHz, throttled=0x0 before and after every run, llama.cpp d73c1d6, -ngl 0, 4 threads, 256 tokens x 2 reps x 2 rounds per point, board power from the Pi 5 PMIC. Five quantizations of the same model, 0 invalid runs. Throughput is the best wait policy per model, which was OMP_WAIT_POLICY=active for all five.
| quant |
file GB |
tok/s |
mean W |
J/token |
GB x tok/s (GB/s) |
| Q2_K |
0.339 |
30.39 |
6.02 |
0.2065 |
10.29 |
| Q3_K_M |
0.355 |
28.39 |
5.66 |
0.2120 |
10.09 |
| Q4_K_M |
0.398 |
25.77 |
6.37 |
0.2585 |
10.25 |
| Q5_K_M |
0.420 |
23.23 |
6.68 |
0.2994 |
9.76 |
| Q8_0 |
0.531 |
19.87 |
5.55 |
0.2924 |
10.55 |
The product of model file size and throughput stayed between 9.76 and 10.55 GB/s across the full sweep, while throughput had a 0.988 correlation with the inverse of model size. I think that is consistent with decode being mainly bandwidth-limited on this particular setup. I would not call it proof of a hard memory-bandwidth ceiling, since file size is only a proxy for the data actually moved and I did not measure an independent STREAM baseline.
I would keep the energy result as a secondary observation rather than make it a second claim in the same issue. Energy per token generally increased as the model got larger, but it was not monotonic: Q8_0 used less energy per token than Q5_K_M. The bandwidth result by itself does not explain that, so I would rather report it without trying to force a conclusion.
My main question is whether a bandwidth-aware metric would be useful in llama-bench for CPU decode. Reporting something like model-size x tokens/second alongside throughput would make it easier to see when a quantization or kernel change is reducing the effective data rate, instead of looking only at token rate.
There are a few limits to this result. It is one Raspberry Pi 5, one model family and one thread count. File size on disk is not the same as bytes streamed per token: it leaves out KV-cache traffic, possible weight repacking and the mixed-precision layout inside the K-quants. The roughly 10 GB/s number should therefore be read as an effective rate, not as measured DRAM bandwidth or a percentage of peak. The full spread is 8.1%, not the roughly 2% I saw in the first three points, and the non-monotonic energy result is another sign that this is not a complete roofline model.
This is related to #25976, but it does not reproduce the same bottleneck. That issue found quantized decode to be compute-bound on a much larger Neoverse-N2 system even though substantial memory bandwidth was still available. My Pi result behaves closer to a bandwidth-bound case, although I cannot compare against peak bandwidth without a STREAM measurement. #25977 is about prefill and GEMM rather than token-by-token decode, so I see this as an additional small-system decode datapoint rather than a confirmation of that result.
I ran the same small dense model across five quantization levels on a Raspberry Pi 5 while keeping the runtime, thread count, clocks and measurement method fixed. I wanted to see whether decode on a small ARM system showed the same kind of memory-wall behavior I have worked on before: as the weight footprint gets larger, does token rate fall in roughly the opposite direction?
Setup: Raspberry Pi 5, 4x Cortex-A76, governor
performanceat 2.4 GHz,throttled=0x0before and after every run, llama.cppd73c1d6,-ngl 0, 4 threads, 256 tokens x 2 reps x 2 rounds per point, board power from the Pi 5 PMIC. Five quantizations of the same model, 0 invalid runs. Throughput is the best wait policy per model, which wasOMP_WAIT_POLICY=activefor all five.The product of model file size and throughput stayed between 9.76 and 10.55 GB/s across the full sweep, while throughput had a 0.988 correlation with the inverse of model size. I think that is consistent with decode being mainly bandwidth-limited on this particular setup. I would not call it proof of a hard memory-bandwidth ceiling, since file size is only a proxy for the data actually moved and I did not measure an independent STREAM baseline.
I would keep the energy result as a secondary observation rather than make it a second claim in the same issue. Energy per token generally increased as the model got larger, but it was not monotonic: Q8_0 used less energy per token than Q5_K_M. The bandwidth result by itself does not explain that, so I would rather report it without trying to force a conclusion.
My main question is whether a bandwidth-aware metric would be useful in
llama-benchfor CPU decode. Reporting something like model-size x tokens/second alongside throughput would make it easier to see when a quantization or kernel change is reducing the effective data rate, instead of looking only at token rate.There are a few limits to this result. It is one Raspberry Pi 5, one model family and one thread count. File size on disk is not the same as bytes streamed per token: it leaves out KV-cache traffic, possible weight repacking and the mixed-precision layout inside the K-quants. The roughly 10 GB/s number should therefore be read as an effective rate, not as measured DRAM bandwidth or a percentage of peak. The full spread is 8.1%, not the roughly 2% I saw in the first three points, and the non-monotonic energy result is another sign that this is not a complete roofline model.
This is related to #25976, but it does not reproduce the same bottleneck. That issue found quantized decode to be compute-bound on a much larger Neoverse-N2 system even though substantial memory bandwidth was still available. My Pi result behaves closer to a bandwidth-bound case, although I cannot compare against peak bandwidth without a STREAM measurement. #25977 is about prefill and GEMM rather than token-by-token decode, so I see this as an additional small-system decode datapoint rather than a confirmation of that result.