Merge pull request #3528 from jasonrandrews/review

tech review of Litespark-Inference install guide

Author: jasonrandrews

content/install-guides/litespark-inference.md

@@ -1,4 +1,18 @@
 ---
+title: Litespark-Inference
+draft: true
+
+description: Install Litespark-Inference on Arm or x86 Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU.
+
+minutes_to_complete: 10
+
+author:
+    - Nii Osae Osae Dade
+    - Tony Morri
+    - Sayandip Pal
+official_docs: https://github.com/Mindbeam-AI/Litespark-Inference
+
+
 additional_search_terms:
 - BitNet
 - ternary
@@ -7,60 +21,65 @@ additional_search_terms:
 - CPU
 - Arm
 - Graviton
-- Apple silicon
+- Apple Silicon
 - Python
 
-layout: installtoolsall
-minutes_to_complete: 10
-author:
-    - Nii Osae Osae Dade
-    - Tony Morri
-    - Sayandip Pal
 multi_install: false
 multitool_install_part: false
-official_docs: https://github.com/Mindbeam-AI/Litespark-Inference
+
 test_images:
 - ubuntu:latest
 test_link: null
 test_maintenance: true
-title: Litespark-Inference
-description: Install Litespark-Inference on Arm or x86 Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU.
+
 tool_install: true
 weight: 1
+layout: installtoolsall
 ---
 
 [Litespark-Inference](https://github.com/Mindbeam-AI/Litespark-Inference)
 is an open-source CPU inference runtime for
 [BitNet b1.58](https://arxiv.org/abs/2402.17764) ternary-weight LLMs. A
 single `pip install` reads your CPU's feature flags and compiles the
-right C++ kernel for it - NEON + SDOT on Arm, AVX-512/VNNI or AVX2+FMA
-on x86. You do not pick the kernel; it picks itself.
+right C++ kernel for it using NEON and SDOT on Arm or AVX-512/VNNI and AVX2+FMA
+on x86. This saves you from needing to pick the right C++ kernel for best performance. 
 
 ## What do I need before installing Litespark-Inference?
 
-- **Python 3.10 or newer.** Check with `python3 --version`.
-- **A C++ toolchain** (`clang` or `g++`). Already present on most
-  developer machines; install steps per platform are below.
-- **About 5 GB of free disk.** The BitNet-2B model downloads from
-  Hugging Face on first run.
+- Python 3.10 or newer, you can run `python3 --version` to check your version
+- A C++ toolchain such as `clang` or `g++`
+- About 5 GB of free disk
+
+The BitNet-2B model is downloaded from Hugging Face on your first run.
 
 It is good practice to install into a clean virtual environment so the
 install does not conflict with anything else on your machine:
 
-```console
+If you don't have Python virtual environment support installed run:
+
+```bash
+sudo apt update
+sudo apt install python3-venv -y
+```
+
+Create a Python virtual environment:
+
+```bash
 python3 -m venv .venv
 source .venv/bin/activate
 python -m pip install --upgrade pip wheel setuptools
 ```
 
-Then follow the section for your platform.
+Next, follow the section for your platform, Linux or macOS.
 
-## How do I install Litespark-Inference on Arm or x86 Linux?
+## How do I install Litespark-Inference on Arm Linux?
 
 The released package builds the correct kernel for your CPU
-automatically - AVX-512 with an AVX2+FMA fallback on x86, and NEON +
+automatically. It uses NEON and 
 SDOT on Arm (Graviton 2/3/4, Ampere, Neoverse N1/N2/V1/V2, Raspberry
-Pi 5). Install the C++ toolchain, then the package:
+Pi 5). 
+
+For Ubuntu/Debian distributions, install the C++ toolchain, then the Litespark-Inference Python package:
 
 ```bash
 sudo apt-get update
@@ -72,6 +91,7 @@ For Red Hat, Fedora, or RHEL, install the toolchain with `dnf` instead:
 
 ```console
 sudo dnf install -y gcc-c++ clang ninja-build git python3-pip
+pip install litespark-inference
 ```
 
 Confirm the package imports and reports the kernel selected for your CPU:
@@ -80,35 +100,42 @@ Confirm the package imports and reports the kernel selected for your CPU:
 python3 -c "import litespark_inference; print(litespark_inference.__version__)"
 ```
 
+The version is printed:
+
+```output
+1.0.3
+```
+
 To inspect which kernel was built, run:
 
-```console
+```bash
 python -m litespark_inference.torchless info
 ```
 
 The output ends with one of the following, depending on your CPU:
 
 ```output
-kernel : avx512 (torchless, extern "C" AVX-512/VNNI + OMP)
-kernel : avx2 (torchless, extern "C" AVX2+FMA fallback + OMP)
-kernel : neon (torchless, extern "C" NEON SDOT)
+litespark_inference.torchless
+  platform : Linux aarch64
+  python   : 3.12.3
+  kernel   : /home/ubuntu/.venv/lib/python3.12/site-packages/litespark_inference/torchless/_matmul_lut_neon.cpython-312-aarch64-linux-gnu.so
+  OpenMP   : True  (max_threads=8)
+  Accelerate: False
 ```
 
 ## How do I install Litespark-Inference on Apple silicon macOS?
 
 Apple's CPUs have NEON SDOT and Litespark-Inference uses it directly.
-The only extra step versus Linux is installing `libomp` - Apple's
+
+The only extra step versus Linux is installing `libomp`, Apple's
 toolchain does not ship a built-in OpenMP runtime, and Litespark uses
-OpenMP for multi-threading inside the kernel:
+OpenMP for multi-threading inside the kernel.
+
+Install Xcode command-line tools, the full Xcode is not required: 
 
 ```console
-# Xcode command-line tools (one-time setup; no full Xcode required)
 xcode-select --install
-
-# libomp via Homebrew
 brew install libomp
-
-# Install the released package from PyPI
 pip install litespark-inference
 ```
 
@@ -118,18 +145,22 @@ Verify the install:
 python -m litespark_inference.torchless info
 ```
 
-Expected output includes:
+The expected output includes:
 
 ```output
-kernel : neon (torchless, extern "C" NEON SDOT)
-OpenMP : True  (max_threads=...)
+litespark_inference.torchless
+  platform : Darwin arm64
+  python   : 3.14.5
+  kernel   : .venv/lib/python3.14/site-packages/litespark_inference/torchless/_matmul_lut_neon.cpython-314-darwin.so
+  OpenMP   : True  (max_threads=12)
+  Accelerate: True
 ```
 
-If `OpenMP : False`, the build did not find Homebrew's `libomp`. The
+If you see `OpenMP : False`, the build did not find Homebrew's `libomp`. The
 most common cause is that Homebrew is installed under `/opt/homebrew`
-(the Apple silicon default) but `pip install` ran in an environment that
-hides it. Re-running `pip install litespark-inference` from a normal
-shell usually fixes it.
+(the Apple Silicon default) but `pip install` ran in an environment that
+hides it. Re-run `pip install litespark-inference` from a normal
+shell to fix it.
 
 ## How do I install from source?
 
@@ -144,14 +175,31 @@ pip install -e .
 
 ## Sanity-check
 
-On all platforms, the same one-liner generates text. The first run
-downloads `microsoft/bitnet-b1.58-2B-4T-bf16` from Hugging Face (around
-4.5 GB into `~/.cache/huggingface/hub`); later runs start instantly:
+On all platforms, the same command can be used. 
+
+The first run downloads the model weights (around 4.5 GB into
+`~/.cache/huggingface/hub`). Subsequent runs do not need to download
+again.
 
 ```console
 litespark-inference generate "Hello, world!" --max-tokens 16
 ```
 
-You are now ready to run BitNet-2B. Continue with the
-[Accelerate LLM inference on Arm CPUs with Litespark-Inference](/learning-paths/laptops-and-desktops/litespark-inference/)
-Learning Path.
+The output is similar to:
+
+```output
+Prompt (36 tokens): 'System: You are Litespark, a helpful AI assistant running locally. Provide accurate, concise, and practical answers.<|eot_id|>User: Hello, world!<|eot_id|>Assistant: '
+Prefill...
+  36 tokens in 0.40s (90.09 tok/s)
+Generate:
+Hello! How can I assist you today?
+
+Generated 9 tokens in 0.29s (31.58 tok/s)
+
+--- output ---
+Hello! How can I assist you today?
+```
+
+You are now ready to run BitNet-2B.
+
+Continue with [Accelerate LLM inference on Arm CPUs with Litespark-Inference](/learning-paths/laptops-and-desktops/litespark-inference/) to learn more.