Skip to content

Commit 8138393

Browse files
jasonrandrewsjasonrandrews
authored
Merge pull request #3191 from annietllnd/gemma-4
WIP LP for Gemma 4
2 parents b5665fc + f23a65f commit 8138393

6 files changed

Lines changed: 337 additions & 0 deletions

File tree

‎content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
---
2+
title: Benchmark Gemma LiteRT-LM prefill performance with KleidiAI and SME2 on macOS
3+
4+
minutes_to_complete: 45
5+
6+
draft: true
7+
cascade:
8+
draft: true
9+
10+
who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS.
11+
12+
learning_objectives:
13+
- Create a pinned workspace with LiteRT-LM, KleidiAI, and XNNPACK
14+
- Install Bazelisk and required macOS prerequisites for LiteRT-LM builds
15+
- Prepare a LiteRT-LM compatible `.litertlm` Gemma model from Hugging Face
16+
- Run LiteRT-LM benchmark commands and measure prefill throughput on a reproducible setup
17+
18+
prerequisites:
19+
- A SME2 device (macOS M4 on Apple Silicon)
20+
- Git, Homebrew, and Xcode Command Line Tools
21+
22+
author: Annie Tallund
23+
24+
### Tags
25+
skilllevels: Advanced
26+
subjects: ML
27+
armips:
28+
- Armv9-A
29+
tools_software_languages:
30+
- SME2
31+
- Bazel
32+
- LiteRT-LM
33+
- KleidiAI
34+
- XNNPACK
35+
operatingsystems:
36+
- macOS
37+
38+
39+
40+
further_reading:
41+
- resource:
42+
title: Arm Scalable Matrix Extension introduction, part 1
43+
link: https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction
44+
type: blog
45+
- resource:
46+
title: Arm Scalable Matrix Extension instructions, part 2
47+
link: https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction-p2
48+
type: blog
49+
- resource:
50+
title: Arm SME2 introduction, part 4
51+
link: https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/part4-arm-sme2-introduction
52+
type: blog
53+
- resource:
54+
title: LiteRT-LM repository
55+
link: https://github.com/google-ai-edge/LiteRT-LM
56+
type: website
57+
- resource:
58+
title: KleidiAI repository
59+
link: https://github.com/ARM-software/kleidiai
60+
type: website
61+
- resource:
62+
title: XNNPACK repository
63+
link: https://github.com/google/xnnpack
64+
type: website
65+
- resource:
66+
title: google/gemma-3n-E4B-it-litert-lm on Hugging Face
67+
link: https://huggingface.co/google/gemma-3n-E4B-it-litert-lm
68+
type: website
69+
- resource:
70+
title: google/gemma-3-4b-it on Hugging Face
71+
link: https://huggingface.co/google/gemma-3-4b-it
72+
type: website
73+
- resource:
74+
title: litert-community/Gemma3-4B-IT on Hugging Face
75+
link: https://huggingface.co/litert-community/Gemma3-4B-IT
76+
type: website
77+
78+
79+
80+
### FIXED, DO NOT MODIFY
81+
# ================================================================================
82+
weight: 1 # _index.md always has weight of 1 to order correctly
83+
layout: "learningpathall" # All files under learning paths have this same wrapper
84+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
85+
---

‎content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_next-steps.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # The weight controls the order of the pages. _index.md always has weight 1.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---

‎content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
title: Install prerequisites and download the model
3+
weight: 4
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Overview
10+
11+
In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma model in LiteRT-LM `.litertlm` format.
12+
13+
## Install prerequisites on macOS
14+
15+
Install Xcode Command Line Tools (if needed):
16+
17+
```bash
18+
xcode-select --install
19+
```
20+
21+
Install Bazelisk:
22+
23+
```bash
24+
brew install bazelisk
25+
```
26+
27+
Install the Hugging Face Hub CLI:
28+
29+
```bash
30+
python3 -m pip install -U "huggingface_hub[cli]"
31+
```
32+
33+
Pin Bazel version `7.6.1` from the LiteRT-LM root:
34+
35+
```bash
36+
cd $HOME/gemma4-prefill-bench/LiteRT-LM
37+
echo "7.6.1" > .bazelversion
38+
bazelisk version
39+
```
40+
41+
## Create model directory
42+
43+
```bash
44+
mkdir -p $HOME/gemma4-prefill-bench/LiteRT-LM/models
45+
```
46+
47+
{{% notice Note %}}
48+
The benchmark commands in this Learning Path assume your model directory is under `LiteRT-LM/models/`.
49+
{{% /notice %}}
50+
51+
## Prepare a LiteRT-LM-compatible model
52+
53+
LiteRT-LM benchmark commands in this Learning Path use `litert_lm_advanced_main`, which expects a LiteRT-LM model artifact (`.litertlm`) for `--model_path`.
54+
55+
The `google/gemma-3-4b-it` repository on Hugging Face follows the Transformers layout (multiple `safetensors` files), so it is not directly consumable by `litert_lm_advanced_main`.
56+
57+
Use a prebuilt LiteRT-LM model artifact instead:
58+
59+
```bash
60+
cd $HOME/gemma4-prefill-bench/LiteRT-LM/models
61+
hf auth login
62+
hf download google/gemma-3n-E4B-it-litert-lm \
63+
--include "gemma-3n-E4B-it-int4.litertlm" \
64+
--local-dir ./gemma-3n-E4B-it-litert-lm
65+
66+
cp ./gemma-3n-E4B-it-litert-lm/gemma-3n-E4B-it-int4.litertlm \
67+
./gemma-3n-E4B-it-int4.litertlm
68+
69+
ls -lh ./gemma-3n-E4B-it-int4.litertlm
70+
```
71+
72+
{{% notice Note %}}
73+
The model repository is gated. Make sure your Hugging Face account has accepted the model terms before downloading.
74+
{{% /notice %}}
75+
76+
Expected layout:
77+
78+
```text
79+
LiteRT-LM/models/
80+
├── gemma-3n-E4B-it-int4.litertlm
81+
└── gemma-3n-E4B-it-litert-lm/
82+
└── gemma-3n-E4B-it-int4.litertlm
83+
```
84+
85+
## What you've accomplished and what's next
86+
87+
In this section:
88+
- You installed macOS prerequisites and pinned Bazel for LiteRT-LM
89+
- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma model
90+
91+
In the next section, you will build and run the benchmark workflow.

‎content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
title: Overview and benchmark workflow
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Why this benchmark matters
10+
11+
This Learning Path gives you a reproducible process to evaluate Gemma prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK.
12+
13+
The flow in this guide is based on a pinned set of repository commits and a local benchmark command (`litert_lm_advanced_main --benchmark`) so your numbers are comparable across runs.
14+
15+
## What you will do
16+
17+
You will complete the workflow in this order:
18+
19+
1. Create a local workspace and clone LiteRT-LM, KleidiAI, and XNNPACK at tested commits.
20+
2. Install macOS prerequisites and Bazelisk, then pin Bazel to version `7.6.1`.
21+
3. Prepare a LiteRT-LM-compatible `.litertlm` model in `LiteRT-LM/models`.
22+
4. Build LiteRT-LM and run benchmark and sample prompts.
23+
24+
In the next section, you will set up the workspace with pinned repository commits.
25+
26+
## Find out if your device supports SME2
27+
28+
Confirm your machine exposes SME/SME2 to user space:
29+
30+
```bash
31+
uname -m
32+
sysctl -a | grep -Ei 'sme2|sme'
33+
```
34+
35+
Expected result:
36+
- `uname -m` should report `arm64`
37+
- `sysctl` output should include an SME/SME2 capability entry
38+
39+
If you do not see SME/SME2 in the `sysctl` output, this benchmark can still run, but XNNPACK/KleidiAI will dispatch non-SME2 kernels and your prefill throughput will likely be lower.
40+
41+
{{% notice Note %}}
42+
For a deeper validation (compiler + runtime streaming mode checks), see the cross-platform Learning Path section [Test your SME2 development environment](./learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment).
43+
{{% /notice %}}

‎content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
title: Build and run benchmarks
3+
weight: 5
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
10+
## Build LiteRT-LM with local XNNPACK and KleidiAI
11+
12+
Build from the LiteRT-LM repository:
13+
14+
```bash
15+
cd $HOME/gemma4-prefill-bench/LiteRT-LM
16+
bazel build \
17+
--config=macos_arm64 \
18+
--macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \
19+
--override_repository=XNNPACK=../xnnpack \
20+
--override_repository=KleidiAI=../kleidiai \
21+
//runtime/engine:litert_lm_advanced_main
22+
23+
```
24+
25+
## Run benchmark
26+
27+
```bash
28+
bazel-bin/runtime/engine/litert_lm_advanced_main \
29+
--backend=cpu \
30+
--model_path=models/gemma-3n-E4B-it-int4.litertlm \
31+
--benchmark \
32+
--benchmark_prefill_tokens=512 \
33+
--benchmark_decode_tokens=128 \
34+
--num_cpu_threads=4 \
35+
--disable_cache=false
36+
```
37+
38+
Example output pattern:
39+
40+
```text
41+
--------------------------------------------------
42+
Time to first token: 0.51 s
43+
--------------------------------------------------
44+
Prefill Turns (Total 1 turns):
45+
Prefill Turn 1: Processed 512 tokens in 477.85ms duration.
46+
Prefill Speed: 1071.47 tokens/sec.
47+
--------------------------------------------------
48+
Decode Turns (Total 1 turns):
49+
Decode Turn 1: Processed 128 tokens in 3.917262s duration.
50+
Decode Speed: 32.68 tokens/sec.
51+
--------------------------------------------------
52+
```
53+
54+
Run quick output sanity checks:
55+
56+
```bash
57+
bazel-bin/runtime/engine/litert_lm_advanced_main \
58+
--backend=cpu \
59+
--model_path=models/gemma-3n-E4B-it-int4.litertlm \
60+
--input_prompt="What is the capital of France?"
61+
62+
bazel-bin/runtime/engine/litert_lm_advanced_main \
63+
--backend=cpu \
64+
--model_path=models/gemma-3n-E4B-it-int4.litertlm \
65+
--input_prompt="What is the most difficult winter Olympic sport?"
66+
```

‎content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
title: Set up workspace and pinned repositories
3+
weight: 3
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Overview
10+
11+
In this section, you prepare a local workspace and clone the exact LiteRT-LM, KleidiAI, and XNNPACK commits used for this benchmark flow.
12+
13+
Create a working directory and move into it:
14+
15+
```bash
16+
mkdir -p $HOME/gemma4-prefill-bench
17+
cd $HOME/gemma4-prefill-bench
18+
```
19+
20+
## Clone tested commits
21+
22+
Clone the repositories and check out pinned commits:
23+
24+
```bash
25+
git clone https://github.com/google-ai-edge/LiteRT-LM.git LiteRT-LM
26+
git -C LiteRT-LM checkout 41d6b964c21f225af9dad7088231b024c369adc1
27+
28+
git clone https://github.com/ARM-software/kleidiai.git kleidiai
29+
git -C kleidiai checkout 72a6a50c1dba714ff27a908e8dd54be3628794b0
30+
31+
git clone https://github.com/google/xnnpack.git xnnpack
32+
git -C xnnpack checkout deb87c027acd2cca4e59fd1f227e523a1640d24c
33+
```
34+
35+
Expected layout:
36+
37+
```text
38+
gemma4-prefill-bench/
39+
├── kleidiai/
40+
├── LiteRT-LM/
41+
└── xnnpack/
42+
```
43+
44+
In the next section, you will install prerequisites and download the model.

0 commit comments

Comments
 (0)