Add final Gemma4 TODOs for hand-over

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md

@@ -1,23 +1,25 @@
 ---
-title: Benchmark Gemma LiteRT-LM prefill performance with KleidiAI and SME2 on macOS
+title: Benchmark Gemma 4 LiteRT-LM prefill performance with KleidiAI and SME2 on macOS
 
 minutes_to_complete: 45
 
 draft: true
 cascade:
     draft: true
 
-who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS.
+who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma 4 prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS.
 
 learning_objectives:
     - Create a pinned workspace with LiteRT-LM, KleidiAI, and XNNPACK
     - Install Bazelisk and required macOS prerequisites for LiteRT-LM builds
-    - Prepare a LiteRT-LM compatible `.litertlm` Gemma model from Hugging Face
-    - Run LiteRT-LM benchmark commands and measure prefill throughput on a reproducible setup
+    - Prepare a LiteRT-LM-compatible `.litertlm` Gemma 4 model from Hugging Face
+    - Run LiteRT-LM benchmark commands and compare prefill throughput with SME2 enabled and disabled
 
 prerequisites:
     - A SME2 device (macOS M4 on Apple Silicon)
     - Git, Homebrew, and Xcode Command Line Tools
+    - At least 25 GB of free disk space for model files and local builds
+
 author: Annie Tallund
 
 generate_summary_faq: true
@@ -64,16 +66,8 @@ further_reading:
         link: https://github.com/google/xnnpack
         type: website
     - resource:
-        title: google/gemma-3n-E4B-it-litert-lm on Hugging Face
-        link: https://huggingface.co/google/gemma-3n-E4B-it-litert-lm
-        type: website
-    - resource:
-        title: google/gemma-3-4b-it on Hugging Face
-        link: https://huggingface.co/google/gemma-3-4b-it
-        type: website
-    - resource:
-        title: litert-community/Gemma3-4B-IT on Hugging Face
-        link: https://huggingface.co/litert-community/Gemma3-4B-IT
+        title: litert-community/gemma-4-E4B-it-litert-lm on Hugging Face
+        link: https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm
         type: website
 
 ### FIXED, DO NOT MODIFY

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md

@@ -8,7 +8,7 @@ layout: learningpathall
 
 ## Overview
 
-In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma model in LiteRT-LM `.litertlm` format.
+In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma 4 model in LiteRT-LM `.litertlm` format.
 
 ## Install prerequisites on macOS
 
@@ -48,44 +48,43 @@ mkdir -p $HOME/gemma4-prefill-bench/LiteRT-LM/models
 The benchmark commands in this Learning Path assume your model directory is under `LiteRT-LM/models/`.
 {{% /notice %}}
 
-## Prepare a LiteRT-LM-compatible model
+## Prepare a LiteRT-LM-compatible Gemma 4 model
 
 LiteRT-LM benchmark commands in this Learning Path use `litert_lm_advanced_main`, which expects a LiteRT-LM model artifact (`.litertlm`) for `--model_path`.
 
-The `google/gemma-3-4b-it` repository on Hugging Face follows the Transformers layout (multiple `safetensors` files), so it is not directly consumable by `litert_lm_advanced_main`.
+The base Gemma model repositories on Hugging Face follow the Transformers layout, so they are not directly consumable by `litert_lm_advanced_main`.
 
 Use a prebuilt LiteRT-LM model artifact instead:
 
 ```bash
 cd $HOME/gemma4-prefill-bench/LiteRT-LM/models
-hf auth login
-hf download google/gemma-3n-E4B-it-litert-lm \
-  --include "gemma-3n-E4B-it-int4.litertlm" \
-  --local-dir ./gemma-3n-E4B-it-litert-lm
+hf download litert-community/gemma-4-E4B-it-litert-lm \
+  gemma-4-E4B-it.litertlm \
+  --local-dir ./gemma-4-E4B-it-litert-lm
 
-cp ./gemma-3n-E4B-it-litert-lm/gemma-3n-E4B-it-int4.litertlm \
-   ./gemma-3n-E4B-it-int4.litertlm
+cp ./gemma-4-E4B-it-litert-lm/gemma-4-E4B-it.litertlm \
+   ./gemma-4-E4B-it.litertlm
 
-ls -lh ./gemma-3n-E4B-it-int4.litertlm
+ls -lh ./gemma-4-E4B-it.litertlm
 ```
 
 {{% notice Note %}}
-The model repository is gated. Make sure your Hugging Face account has accepted the model terms before downloading.
+If the download is denied, run `hf auth login`, accept the model terms in Hugging Face, and repeat the download command.
 {{% /notice %}}
 
 Expected layout:
 
 ```text
 LiteRT-LM/models/
-├── gemma-3n-E4B-it-int4.litertlm
-└── gemma-3n-E4B-it-litert-lm/
-    └── gemma-3n-E4B-it-int4.litertlm
+├── gemma-4-E4B-it.litertlm
+└── gemma-4-E4B-it-litert-lm/
+    └── gemma-4-E4B-it.litertlm
 ```
 
 ## What you've accomplished and what's next
 
 In this section:
 - You installed macOS prerequisites and pinned Bazel for LiteRT-LM
-- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma model
+- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma 4 model
 
 In the next section, you will build and run the benchmark workflow.

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md

@@ -8,7 +8,7 @@ layout: learningpathall
 
 ## Why this benchmark matters
 
-This Learning Path gives you a reproducible process to evaluate Gemma prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK.
+This Learning Path gives you a reproducible process to evaluate Gemma 4 prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK.
 
 The flow in this guide is based on a pinned set of repository commits and a local benchmark command (`litert_lm_advanced_main --benchmark`) so your numbers are comparable across runs.
 
@@ -18,8 +18,8 @@ You will complete the workflow in this order:
 
 1. Create a local workspace and clone LiteRT-LM, KleidiAI, and XNNPACK at tested commits.
 2. Install macOS prerequisites and Bazelisk, then pin Bazel to version `7.6.1`.
-3. Prepare a LiteRT-LM-compatible `.litertlm` model in `LiteRT-LM/models`.
-4. Build LiteRT-LM and run benchmark and sample prompts.
+3. Prepare a LiteRT-LM-compatible Gemma 4 `.litertlm` model in `LiteRT-LM/models`.
+4. Build LiteRT-LM with SME2 enabled and disabled, then compare benchmark results.
 
 In the next section, you will set up the workspace with pinned repository commits.
 

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md

@@ -9,30 +9,71 @@ layout: learningpathall
 
 ## Build LiteRT-LM with local XNNPACK and KleidiAI
 
-Build from the LiteRT-LM repository:
+Build the SME2-enabled benchmark binary from the LiteRT-LM repository:
 
 ```bash
 cd $HOME/gemma4-prefill-bench/LiteRT-LM
 bazel build \
     --config=macos_arm64 \
+    --define=xnn_enable_arm_sme=true \
+    --define=xnn_enable_arm_sme2=true \
+    --define=xnn_enable_kleidiai=true \
     --macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \
     --override_repository=XNNPACK=../xnnpack \
     --override_repository=KleidiAI=../kleidiai \
     //runtime/engine:litert_lm_advanced_main
 
+cp bazel-bin/runtime/engine/litert_lm_advanced_main \
+   ./litert_lm_advanced_main_sme2_on
 ```
 
-## Run benchmark
+Build a second binary with KleidiAI SME and SME2 paths disabled so you can compare the same workload against the XNNPACK fallback on the same machine:
 
 ```bash
-bazel-bin/runtime/engine/litert_lm_advanced_main \
+bazel build \
+    --config=macos_arm64 \
+    --define=xnn_enable_arm_sme=false \
+    --define=xnn_enable_arm_sme2=false \
+    --define=xnn_enable_kleidiai=false \
+    --macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \
+    --override_repository=XNNPACK=../xnnpack \
+    --override_repository=KleidiAI=../kleidiai \
+    //runtime/engine:litert_lm_advanced_main
+
+cp bazel-bin/runtime/engine/litert_lm_advanced_main \
+   ./litert_lm_advanced_main_sme2_off
+```
+
+{{% notice Note %}}
+TODO before publication: rerun both binaries on the final pinned upstream XNNPACK and KleidiAI commits, then replace the sample benchmark output with measured Gemma 4 numbers from the same SME2-capable macOS system.
+{{% /notice %}}
+
+## Run benchmarks with and without SME2
+
+Run the SME2-enabled benchmark:
+
+```bash
+./litert_lm_advanced_main_sme2_on \
   --backend=cpu \
-  --model_path=models/gemma-3n-E4B-it-int4.litertlm \
+  --model_path=models/gemma-4-E4B-it.litertlm \
   --benchmark \
   --benchmark_prefill_tokens=512 \
   --benchmark_decode_tokens=128 \
   --num_cpu_threads=4 \
-  --disable_cache=false
+  --disable_cache=false | tee benchmark-sme2-on.txt
+```
+
+Run the SME2-disabled benchmark:
+
+```bash
+./litert_lm_advanced_main_sme2_off \
+  --backend=cpu \
+  --model_path=models/gemma-4-E4B-it.litertlm \
+  --benchmark \
+  --benchmark_prefill_tokens=512 \
+  --benchmark_decode_tokens=128 \
+  --num_cpu_threads=4 \
+  --disable_cache=false | tee benchmark-sme2-off.txt
 ```
 
 Example output pattern:
@@ -51,16 +92,36 @@ Example output pattern:
 --------------------------------------------------
 ```
 
+## Verify the SME2 performance uplift
+
+Compare the prefill throughput from both runs:
+
+```bash
+SME2_ON=$(awk '/Prefill Speed/ {print $3; exit}' benchmark-sme2-on.txt)
+SME2_OFF=$(awk '/Prefill Speed/ {print $3; exit}' benchmark-sme2-off.txt)
+
+SME2_ON="$SME2_ON" SME2_OFF="$SME2_OFF" \
+python3 -c 'import os; on=float(os.environ["SME2_ON"]); off=float(os.environ["SME2_OFF"]); print(f"SME2 prefill uplift: {on/off:.2f}x ({off:.2f} -> {on:.2f} tokens/sec)")'
+```
+
+For a reliable comparison:
+
+- Use the same model, token counts, thread count, power state, and terminal session for both runs.
+- Run each binary at least three times and compare the median `Prefill Speed`.
+- Treat decode throughput separately. This Learning Path focuses on prefill because the SME2-optimized matrix multiplication path has the clearest effect during the prompt-processing phase.
+
+## Run sample prompts
+
 Run quick output sanity checks:
 
 ```bash
-bazel-bin/runtime/engine/litert_lm_advanced_main \
+./litert_lm_advanced_main_sme2_on \
   --backend=cpu \
-  --model_path=models/gemma-3n-E4B-it-int4.litertlm \
+  --model_path=models/gemma-4-E4B-it.litertlm \
   --input_prompt="What is the capital of France?"
 
-bazel-bin/runtime/engine/litert_lm_advanced_main \
+./litert_lm_advanced_main_sme2_on \
   --backend=cpu \
-  --model_path=models/gemma-3n-E4B-it-int4.litertlm \
+  --model_path=models/gemma-4-E4B-it.litertlm \
   --input_prompt="What is the most difficult winter Olympic sport?"
 ```

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md

@@ -10,6 +10,10 @@ layout: learningpathall
 
 In this section, you prepare a local workspace and clone the exact LiteRT-LM, KleidiAI, and XNNPACK commits used for this benchmark flow.
 
+{{% notice Note %}}
+TODO before publication: replace the pinned KleidiAI and XNNPACK commits below with the final known-good upstream commits after the LiteRT-LM dependency update lands. If LiteRT-LM consumes those upstream revisions directly, remove the local `--override_repository` flags in the build commands and simplify this section to clone only LiteRT-LM.
+{{% /notice %}}
+
 Create a working directory and move into it:
 
 ```bash