Skip to content

et-backend: F16 decode GEMV rewrite for register-resident vecdot#12

Closed
RehanQasim-dev wants to merge 1 commit into
etfrom
feat/f16-vecdot-decode
Closed

et-backend: F16 decode GEMV rewrite for register-resident vecdot#12
RehanQasim-dev wants to merge 1 commit into
etfrom
feat/f16-vecdot-decode

Conversation

@RehanQasim-dev

@RehanQasim-dev RehanQasim-dev commented Jul 14, 2026

Copy link
Copy Markdown

Summary

  • Stripes output elements across every hart of all 32 shires instead of blocking work into 16-element chunks, which only filled 8 shires for a typical decode GEMV.
  • Adds a register-resident f16 row-dot helper and stages the reused B activation vector into per-shire L2 SCP so it survives weight-matrix streaming.

Performance (Llama-3.2-1B-Instruct, ET-SoC-1)

Both prefill and decode improved roughly 2x.

Prefill t/s

N et optimized speedup
100 38.6 75.6 1.96x
220 39.8 77.9 1.96x
512 39.0 76.1 1.95x
700 39.2 76.1 1.94x
900 39.1 75.9 1.94x

Decode t/s (@context=32)

tg et optimized speedup
15 4.23 9.49 2.24x
30 4.23 9.49 2.24x
55 4.22 9.44 2.24x
85 4.22 9.46 2.24x
120 4.22 9.45 2.24x

Testing

Verified with llama-cli on ET-SoC-1 hardware (Llama-3.2-1B-Instruct), coherent output.

Stripe output elements across every hart of all 32 shires instead of
blocking work into 16-element chunks, which only filled 8 shires for a
typical decode GEMV. Adds a register-resident f16 row-dot helper and
stages the reused B activation vector into per-shire L2 SCP so it
survives weight-matrix streaming.
@github-actions github-actions Bot added the ggml label Jul 14, 2026
@RehanQasim-dev

Copy link
Copy Markdown
Author

Superseded by the upstream PR against ggml-org/llama.cpp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant