Merge pull request #3471 from tonymindbeamai/learning-path/litespark-inference

Add Learning Path: Accelerate LLM inference on Arm CPUs with Litespark-Inference

Author: jasonrandrews

assets/contributors.csv

@@ -2,7 +2,7 @@ author,company,github,linkedin,twitter,website
 Jason Andrews,Arm,jasonrandrews,jason-andrews-7b05a8,,
 Doug Anson,Arm,DougAnsonAustinTx,douganson,,
 Pareena Verma,Arm,pareenaverma,pareena-verma-7853607,,
-Jaidev Singh Chadha,Arm,jaidev17,jaidevsinghchadha,,
+Jaidev Singh Chadha,Arm,jaidev17,jaidevsinghchadha,,
 Ronan Synnott,Arm,,ronansynnott,,
 Florent Lebeau,Arm,,,,
 Brenda Strech,Remote.It,bstrech,bstrech,@remote_it,www.remote.it
@@ -89,7 +89,7 @@ Geremy Cohen,Arm,geremyCohen,geremyinanutshell,,
 Barbara Corriero,Arm,,,,
 Nina Drozd,Arm,NinaARM,ninadrozd,,
 Jun He,Arm,JunHe77,jun-he-91969822,,
-Henry Wang,Arm,MrXinWang,xin-wang-930b4b141,,
+Henry Wang,Arm,MrXinWang,xin-wang-930b4b141,,
 Gian Marco Iodice,Arm,,,,
 Aude Vuilliomenet,Arm,,,,
 Andrew Kilroy,Arm,,,,
@@ -131,4 +131,7 @@ Anupras Mohapatra,Arm,,,,
 Tomas Agustin Gonzalez Orlando,Arm,tgonzalezorlandoarm,tgorlando,,
 Jai Adam Schrem,Arm,jai-adam-schrem,jai-adam-s-395bb1233,,
 Steve Jordahl,Rafay,,,,
-Dave Neary,Ampere,,,,
+Nii Osae Osae Dade,Mindbeam,StarkOsae,,,
+Tony Morri,Mindbeam,tonymindbeamai,,,
+Sayandip Pal,Mindbeam,sayandip-pal,,,
+Dave Neary,Ampere,,,,

content/install-guides/litespark-inference.md

@@ -0,0 +1,157 @@
+---
+additional_search_terms:
+- BitNet
+- ternary
+- LLM
+- inference
+- CPU
+- Arm
+- Graviton
+- Apple silicon
+- Python
+
+layout: installtoolsall
+minutes_to_complete: 10
+author:
+    - Nii Osae Osae Dade
+    - Tony Morri
+    - Sayandip Pal
+multi_install: false
+multitool_install_part: false
+official_docs: https://github.com/Mindbeam-AI/Litespark-Inference
+test_images:
+- ubuntu:latest
+test_link: null
+test_maintenance: true
+title: Litespark-Inference
+description: Install Litespark-Inference on Arm or x86 Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU.
+tool_install: true
+weight: 1
+---
+
+[Litespark-Inference](https://github.com/Mindbeam-AI/Litespark-Inference)
+is an open-source CPU inference runtime for
+[BitNet b1.58](https://arxiv.org/abs/2402.17764) ternary-weight LLMs. A
+single `pip install` reads your CPU's feature flags and compiles the
+right C++ kernel for it - NEON + SDOT on Arm, AVX-512/VNNI or AVX2+FMA
+on x86. You do not pick the kernel; it picks itself.
+
+## What do I need before installing Litespark-Inference?
+
+- **Python 3.10 or newer.** Check with `python3 --version`.
+- **A C++ toolchain** (`clang` or `g++`). Already present on most
+  developer machines; install steps per platform are below.
+- **About 5 GB of free disk.** The BitNet-2B model downloads from
+  Hugging Face on first run.
+
+It is good practice to install into a clean virtual environment so the
+install does not conflict with anything else on your machine:
+
+```console
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip wheel setuptools
+```
+
+Then follow the section for your platform.
+
+## How do I install Litespark-Inference on Arm or x86 Linux?
+
+The released package builds the correct kernel for your CPU
+automatically - AVX-512 with an AVX2+FMA fallback on x86, and NEON +
+SDOT on Arm (Graviton 2/3/4, Ampere, Neoverse N1/N2/V1/V2, Raspberry
+Pi 5). Install the C++ toolchain, then the package:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y build-essential clang ninja-build git python3-pip
+pip install litespark-inference
+```
+
+For Red Hat, Fedora, or RHEL, install the toolchain with `dnf` instead:
+
+```console
+sudo dnf install -y gcc-c++ clang ninja-build git python3-pip
+```
+
+Confirm the package imports and reports the kernel selected for your CPU:
+
+```bash
+python3 -c "import litespark_inference; print(litespark_inference.__version__)"
+```
+
+To inspect which kernel was built, run:
+
+```console
+python -m litespark_inference.torchless info
+```
+
+The output ends with one of the following, depending on your CPU:
+
+```output
+kernel : avx512 (torchless, extern "C" AVX-512/VNNI + OMP)
+kernel : avx2 (torchless, extern "C" AVX2+FMA fallback + OMP)
+kernel : neon (torchless, extern "C" NEON SDOT)
+```
+
+## How do I install Litespark-Inference on Apple silicon macOS?
+
+Apple's CPUs have NEON SDOT and Litespark-Inference uses it directly.
+The only extra step versus Linux is installing `libomp` - Apple's
+toolchain does not ship a built-in OpenMP runtime, and Litespark uses
+OpenMP for multi-threading inside the kernel:
+
+```console
+# Xcode command-line tools (one-time setup; no full Xcode required)
+xcode-select --install
+
+# libomp via Homebrew
+brew install libomp
+
+# Install the released package from PyPI
+pip install litespark-inference
+```
+
+Verify the install:
+
+```console
+python -m litespark_inference.torchless info
+```
+
+Expected output includes:
+
+```output
+kernel : neon (torchless, extern "C" NEON SDOT)
+OpenMP : True  (max_threads=...)
+```
+
+If `OpenMP : False`, the build did not find Homebrew's `libomp`. The
+most common cause is that Homebrew is installed under `/opt/homebrew`
+(the Apple silicon default) but `pip install` ran in an environment that
+hides it. Re-running `pip install litespark-inference` from a normal
+shell usually fixes it.
+
+## How do I install from source?
+
+To modify the runtime or kernels, install from source instead of from
+PyPI:
+
+```console
+git clone https://github.com/Mindbeam-AI/Litespark-Inference.git
+cd Litespark-Inference
+pip install -e .
+```
+
+## Sanity-check
+
+On all platforms, the same one-liner generates text. The first run
+downloads `microsoft/bitnet-b1.58-2B-4T-bf16` from Hugging Face (around
+4.5 GB into `~/.cache/huggingface/hub`); later runs start instantly:
+
+```console
+litespark-inference generate "Hello, world!" --max-tokens 16
+```
+
+You are now ready to run BitNet-2B. Continue with the
+[Accelerate LLM inference on Arm CPUs with Litespark-Inference](/learning-paths/laptops-and-desktops/litespark-inference/)
+Learning Path.

content/learning-paths/laptops-and-desktops/litespark-inference/01-introduction.md

@@ -0,0 +1,73 @@
+---
+title: What is Litespark-Inference?
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## The short version
+
+Litespark-Inference is an open-source runtime that runs
+ternary-weight language models - such as
+[BitNet b1.58](https://arxiv.org/abs/2402.17764) - on the CPU you
+already have.
+No GPU. No CUDA wheel. No PyTorch installation. The dependency
+footprint after `pip install` is `numpy`, `safetensors`, and
+`tokenizers`.
+
+It runs on:
+
+| Platform | Kernel selected automatically |
+|---|---|
+| **Linux on x86_64** with AVX-512 (Intel Ice Lake+ / AMD Zen 4+) | AVX-512 + VNNI |
+| **Linux on x86_64**, Intel Core Ultra (no AVX-512) | AVX-VNNI (256-bit) |
+| **Linux on x86_64** without VNNI (AMD Zen 2-3, pre-Skylake-X Intel) | AVX2 + FMA fallback |
+| **Linux on Arm (Graviton 2/3/4, Ampere, etc.)** | NEON + SDOT |
+| **macOS on Apple silicon (M1/M2/M3/M4/M5)** | NEON + SDOT |
+
+A single `pip install` reads your CPU's feature flags and compiles the
+right C++ source for you. You don't pick the kernel; it picks itself.
+The [Litespark-Inference install guide](/install-guides/litespark-inference/)
+covers the install step for each platform.
+
+## Why BitNet on the CPU
+
+BitNet b1.58 stores each weight as a value in `{-1, 0, +1}` and packs
+four weights into one byte. That means:
+
+- The model file is around 6x smaller than the equivalent `bfloat16`
+  model (around 497 MB packed versus around 4,600 MB unpacked).
+- Every matmul reduces to `int8` activation x ternary weight - exactly
+  what a CPU's SIMD dot-product instructions (SDOT on Arm, VNNI on
+  x86) are designed for.
+
+The net effect: a 2-billion-parameter model that fits in under 1 GB of
+RAM and generates tokens at interactive speed on a normal laptop or
+cloud CPU instance.
+
+The charts below show Litespark-Inference against a PyTorch baseline
+across several Arm and x86 CPUs. Token-generation throughput is roughly
+an order of magnitude higher, and resident memory is around 6x smaller,
+on every platform tested.
+
+![Token-generation throughput, Litespark-Inference versus PyTorch, on Apple M5 Max, AMD Zen 4, and Intel Core Ultra 9#center](throughput-comparison.png "Cross-platform throughput comparison")
+
+![Resident memory, Litespark-Inference versus PyTorch, on Apple M5 Max, AMD Zen 4, and Intel Core Ultra 9#center](memory-comparison.png "Cross-platform memory usage comparison")
+
+## What you'll do in this Learning Path
+
+1. **Run BitNet-2B from the CLI in one line:**
+   `litespark-inference generate "Why is BitNet fast on CPU?" --max-tokens 64`.
+
+2. **Run the same model from a short Python script** using the
+   high-level `BitNet.from_pretrained(...).generate(...)` API.
+
+3. **Tune the embedding dtype** (`bf16` / `int8` / `int4`) to trade
+   memory for quality.
+
+4. **(Optional)** Run a head-to-head benchmark against `transformers`
+   with `torch.bfloat16` on the same machine, with memory, TTFT,
+   throughput, and energy-per-token numbers.
+
+The next chapter runs BitNet-2B from the CLI and from Python.

content/learning-paths/laptops-and-desktops/litespark-inference/02-run.md

@@ -0,0 +1,93 @@
+---
+title: Run BitNet-2B
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+With Litespark-Inference installed (see the
+[install guide](/install-guides/litespark-inference/)), you can generate
+text right away.
+
+## The CLI (one line)
+
+```bash
+litespark-inference generate "Why is BitNet fast on CPU?" --max-tokens 64
+```
+
+What this does:
+
+1. Loads `microsoft/bitnet-b1.58-2B-4T-bf16` from the Hugging Face cache
+   (downloaded on first run, around 4.5 GB).
+2. Tokenizes the prompt and runs it through the model in one batched
+   **prefill** call.
+3. Greedy-decodes 64 new tokens, streaming each one to your terminal
+   as it is produced.
+
+Other useful CLI commands:
+
+```bash
+# Quick health-check - what kernel, what threads, what model
+litespark-inference info
+
+# Interactive chat (multi-turn, reuses KV cache across turns)
+litespark-inference chat
+```
+
+## From Python (a few lines)
+
+If you want to embed Litespark-Inference in an application, the
+recommended API is the high-level `BitNet` class - load the model, then
+call `generate`:
+
+```python
+from litespark_inference.torchless import BitNet
+
+# Auto-downloads from Hugging Face on first use.
+bn = BitNet.from_pretrained("bitnet-2b")
+
+# chat=True applies the system + chat template and returns a clean
+# instruction-following answer (omit it for a raw continuation).
+print(bn.generate("Why is BitNet fast on CPU?", max_new_tokens=64, chat=True))
+```
+
+That is the whole surface: `from_pretrained` loads the packed model and
+owns the KV cache; `generate` runs the batched prefill plus
+autoregressive decoding for you. There is no `torch` import at inference.
+
+## Embed dtype: pick your memory / quality trade-off
+
+The weight matrices are always 2-bit packed (that is what BitNet b1.58
+means). The one thing you can vary is the token-embedding table's
+dtype - and that controls a big chunk of the model's resident memory:
+
+| Embed dtype | Resident memory | Quality | When to use |
+|---|---|---|---|
+| `bf16` | around 813 MB | Reference | Default for accuracy-sensitive work |
+| `int8` | around 656 MB | Indistinguishable from `bf16` in practice | Good balance |
+| **`int4`** | **around 573 MB** | Slight quality cost on rare tokens | **Recommended** - fastest load, smallest footprint |
+
+CLI:
+
+```bash
+litespark-inference generate "Why is BitNet fast on CPU?" --embed-dtype int4 --max-tokens 64
+```
+
+Python:
+
+```python
+from litespark_inference.torchless import BitNet
+
+bn = BitNet.from_pretrained("bitnet-2b", embed_dtype="int4")
+print(bn.generate("Why is BitNet fast on CPU?", max_new_tokens=64, chat=True))
+```
+
+## What's next
+
+If you just wanted to run BitNet on the CPU, you are done - go build
+something with it.
+
+If you want to **know how fast it actually is** versus PyTorch on the
+same machine (and how much energy it uses), continue to the next chapter
+on benchmarking.