diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md index 58f670584b..aabff9ae1b 100755 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md @@ -1,5 +1,5 @@ --- -title: Benchmark Gemma LiteRT-LM prefill performance with KleidiAI and SME2 on macOS +title: Benchmark Gemma 4 LiteRT-LM prefill performance with KleidiAI and SME2 on macOS minutes_to_complete: 45 @@ -7,17 +7,19 @@ draft: true cascade: draft: true -who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS. +who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma 4 prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS. learning_objectives: - Create a pinned workspace with LiteRT-LM, KleidiAI, and XNNPACK - Install Bazelisk and required macOS prerequisites for LiteRT-LM builds - - Prepare a LiteRT-LM compatible `.litertlm` Gemma model from Hugging Face - - Run LiteRT-LM benchmark commands and measure prefill throughput on a reproducible setup + - Prepare a LiteRT-LM-compatible `.litertlm` Gemma 4 model from Hugging Face + - Run LiteRT-LM benchmark commands and compare prefill throughput with SME2 enabled and disabled prerequisites: - A SME2 device (macOS M4 on Apple Silicon) - Git, Homebrew, and Xcode Command Line Tools + - At least 25 GB of free disk space for model files and local builds + author: Annie Tallund generate_summary_faq: true @@ -64,16 +66,8 @@ further_reading: link: https://github.com/google/xnnpack type: website - resource: - title: google/gemma-3n-E4B-it-litert-lm on Hugging Face - link: https://huggingface.co/google/gemma-3n-E4B-it-litert-lm - type: website - - resource: - title: google/gemma-3-4b-it on Hugging Face - link: https://huggingface.co/google/gemma-3-4b-it - type: website - - resource: - title: litert-community/Gemma3-4B-IT on Hugging Face - link: https://huggingface.co/litert-community/Gemma3-4B-IT + title: litert-community/gemma-4-E4B-it-litert-lm on Hugging Face + link: https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm type: website ### FIXED, DO NOT MODIFY diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md index dfb041af0b..b728662e06 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md @@ -8,7 +8,7 @@ layout: learningpathall ## Overview -In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma model in LiteRT-LM `.litertlm` format. +In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma 4 model in LiteRT-LM `.litertlm` format. ## Install prerequisites on macOS @@ -48,44 +48,43 @@ mkdir -p $HOME/gemma4-prefill-bench/LiteRT-LM/models The benchmark commands in this Learning Path assume your model directory is under `LiteRT-LM/models/`. {{% /notice %}} -## Prepare a LiteRT-LM-compatible model +## Prepare a LiteRT-LM-compatible Gemma 4 model LiteRT-LM benchmark commands in this Learning Path use `litert_lm_advanced_main`, which expects a LiteRT-LM model artifact (`.litertlm`) for `--model_path`. -The `google/gemma-3-4b-it` repository on Hugging Face follows the Transformers layout (multiple `safetensors` files), so it is not directly consumable by `litert_lm_advanced_main`. +The base Gemma model repositories on Hugging Face follow the Transformers layout, so they are not directly consumable by `litert_lm_advanced_main`. Use a prebuilt LiteRT-LM model artifact instead: ```bash cd $HOME/gemma4-prefill-bench/LiteRT-LM/models -hf auth login -hf download google/gemma-3n-E4B-it-litert-lm \ - --include "gemma-3n-E4B-it-int4.litertlm" \ - --local-dir ./gemma-3n-E4B-it-litert-lm +hf download litert-community/gemma-4-E4B-it-litert-lm \ + gemma-4-E4B-it.litertlm \ + --local-dir ./gemma-4-E4B-it-litert-lm -cp ./gemma-3n-E4B-it-litert-lm/gemma-3n-E4B-it-int4.litertlm \ - ./gemma-3n-E4B-it-int4.litertlm +cp ./gemma-4-E4B-it-litert-lm/gemma-4-E4B-it.litertlm \ + ./gemma-4-E4B-it.litertlm -ls -lh ./gemma-3n-E4B-it-int4.litertlm +ls -lh ./gemma-4-E4B-it.litertlm ``` {{% notice Note %}} -The model repository is gated. Make sure your Hugging Face account has accepted the model terms before downloading. +If the download is denied, run `hf auth login`, accept the model terms in Hugging Face, and repeat the download command. {{% /notice %}} Expected layout: ```text LiteRT-LM/models/ -├── gemma-3n-E4B-it-int4.litertlm -└── gemma-3n-E4B-it-litert-lm/ - └── gemma-3n-E4B-it-int4.litertlm +├── gemma-4-E4B-it.litertlm +└── gemma-4-E4B-it-litert-lm/ + └── gemma-4-E4B-it.litertlm ``` ## What you've accomplished and what's next In this section: - You installed macOS prerequisites and pinned Bazel for LiteRT-LM -- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma model +- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma 4 model In the next section, you will build and run the benchmark workflow. diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md index b8617cd494..ef8d26cf41 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md @@ -8,7 +8,7 @@ layout: learningpathall ## Why this benchmark matters -This Learning Path gives you a reproducible process to evaluate Gemma prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK. +This Learning Path gives you a reproducible process to evaluate Gemma 4 prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK. The flow in this guide is based on a pinned set of repository commits and a local benchmark command (`litert_lm_advanced_main --benchmark`) so your numbers are comparable across runs. @@ -18,8 +18,8 @@ You will complete the workflow in this order: 1. Create a local workspace and clone LiteRT-LM, KleidiAI, and XNNPACK at tested commits. 2. Install macOS prerequisites and Bazelisk, then pin Bazel to version `7.6.1`. -3. Prepare a LiteRT-LM-compatible `.litertlm` model in `LiteRT-LM/models`. -4. Build LiteRT-LM and run benchmark and sample prompts. +3. Prepare a LiteRT-LM-compatible Gemma 4 `.litertlm` model in `LiteRT-LM/models`. +4. Build LiteRT-LM with SME2 enabled and disabled, then compare benchmark results. In the next section, you will set up the workspace with pinned repository commits. diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md index 71eee0f472..ec405da5e7 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md @@ -9,30 +9,71 @@ layout: learningpathall ## Build LiteRT-LM with local XNNPACK and KleidiAI -Build from the LiteRT-LM repository: +Build the SME2-enabled benchmark binary from the LiteRT-LM repository: ```bash cd $HOME/gemma4-prefill-bench/LiteRT-LM bazel build \ --config=macos_arm64 \ + --define=xnn_enable_arm_sme=true \ + --define=xnn_enable_arm_sme2=true \ + --define=xnn_enable_kleidiai=true \ --macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \ --override_repository=XNNPACK=../xnnpack \ --override_repository=KleidiAI=../kleidiai \ //runtime/engine:litert_lm_advanced_main +cp bazel-bin/runtime/engine/litert_lm_advanced_main \ + ./litert_lm_advanced_main_sme2_on ``` -## Run benchmark +Build a second binary with KleidiAI SME and SME2 paths disabled so you can compare the same workload against the XNNPACK fallback on the same machine: ```bash -bazel-bin/runtime/engine/litert_lm_advanced_main \ +bazel build \ + --config=macos_arm64 \ + --define=xnn_enable_arm_sme=false \ + --define=xnn_enable_arm_sme2=false \ + --define=xnn_enable_kleidiai=false \ + --macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \ + --override_repository=XNNPACK=../xnnpack \ + --override_repository=KleidiAI=../kleidiai \ + //runtime/engine:litert_lm_advanced_main + +cp bazel-bin/runtime/engine/litert_lm_advanced_main \ + ./litert_lm_advanced_main_sme2_off +``` + +{{% notice Note %}} +TODO before publication: rerun both binaries on the final pinned upstream XNNPACK and KleidiAI commits, then replace the sample benchmark output with measured Gemma 4 numbers from the same SME2-capable macOS system. +{{% /notice %}} + +## Run benchmarks with and without SME2 + +Run the SME2-enabled benchmark: + +```bash +./litert_lm_advanced_main_sme2_on \ --backend=cpu \ - --model_path=models/gemma-3n-E4B-it-int4.litertlm \ + --model_path=models/gemma-4-E4B-it.litertlm \ --benchmark \ --benchmark_prefill_tokens=512 \ --benchmark_decode_tokens=128 \ --num_cpu_threads=4 \ - --disable_cache=false + --disable_cache=false | tee benchmark-sme2-on.txt +``` + +Run the SME2-disabled benchmark: + +```bash +./litert_lm_advanced_main_sme2_off \ + --backend=cpu \ + --model_path=models/gemma-4-E4B-it.litertlm \ + --benchmark \ + --benchmark_prefill_tokens=512 \ + --benchmark_decode_tokens=128 \ + --num_cpu_threads=4 \ + --disable_cache=false | tee benchmark-sme2-off.txt ``` Example output pattern: @@ -51,16 +92,36 @@ Example output pattern: -------------------------------------------------- ``` +## Verify the SME2 performance uplift + +Compare the prefill throughput from both runs: + +```bash +SME2_ON=$(awk '/Prefill Speed/ {print $3; exit}' benchmark-sme2-on.txt) +SME2_OFF=$(awk '/Prefill Speed/ {print $3; exit}' benchmark-sme2-off.txt) + +SME2_ON="$SME2_ON" SME2_OFF="$SME2_OFF" \ +python3 -c 'import os; on=float(os.environ["SME2_ON"]); off=float(os.environ["SME2_OFF"]); print(f"SME2 prefill uplift: {on/off:.2f}x ({off:.2f} -> {on:.2f} tokens/sec)")' +``` + +For a reliable comparison: + +- Use the same model, token counts, thread count, power state, and terminal session for both runs. +- Run each binary at least three times and compare the median `Prefill Speed`. +- Treat decode throughput separately. This Learning Path focuses on prefill because the SME2-optimized matrix multiplication path has the clearest effect during the prompt-processing phase. + +## Run sample prompts + Run quick output sanity checks: ```bash -bazel-bin/runtime/engine/litert_lm_advanced_main \ +./litert_lm_advanced_main_sme2_on \ --backend=cpu \ - --model_path=models/gemma-3n-E4B-it-int4.litertlm \ + --model_path=models/gemma-4-E4B-it.litertlm \ --input_prompt="What is the capital of France?" -bazel-bin/runtime/engine/litert_lm_advanced_main \ +./litert_lm_advanced_main_sme2_on \ --backend=cpu \ - --model_path=models/gemma-3n-E4B-it-int4.litertlm \ + --model_path=models/gemma-4-E4B-it.litertlm \ --input_prompt="What is the most difficult winter Olympic sport?" ``` diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md index 8221122ea8..29e1e64eca 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md @@ -10,6 +10,10 @@ layout: learningpathall In this section, you prepare a local workspace and clone the exact LiteRT-LM, KleidiAI, and XNNPACK commits used for this benchmark flow. +{{% notice Note %}} +TODO before publication: replace the pinned KleidiAI and XNNPACK commits below with the final known-good upstream commits after the LiteRT-LM dependency update lands. If LiteRT-LM consumes those upstream revisions directly, remove the local `--override_repository` flags in the build commands and simplify this section to clone only LiteRT-LM. +{{% /notice %}} + Create a working directory and move into it: ```bash