From 2f7191ba1b29c0628f932cae94b6ed96cc5af15a Mon Sep 17 00:00:00 2001 From: Jason Andrews Date: Fri, 24 Jul 2026 16:02:15 -0500 Subject: [PATCH] tech review of Litespark-Inference install guide --- content/install-guides/litespark-inference.md | 140 ++++++++++++------ 1 file changed, 94 insertions(+), 46 deletions(-) diff --git a/content/install-guides/litespark-inference.md b/content/install-guides/litespark-inference.md index d50828f1a3..8905d8f1bb 100644 --- a/content/install-guides/litespark-inference.md +++ b/content/install-guides/litespark-inference.md @@ -1,4 +1,18 @@ --- +title: Litespark-Inference +draft: true + +description: Install Litespark-Inference on Arm or x86 Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU. + +minutes_to_complete: 10 + +author: + - Nii Osae Osae Dade + - Tony Morri + - Sayandip Pal +official_docs: https://github.com/Mindbeam-AI/Litespark-Inference + + additional_search_terms: - BitNet - ternary @@ -7,60 +21,65 @@ additional_search_terms: - CPU - Arm - Graviton -- Apple silicon +- Apple Silicon - Python -layout: installtoolsall -minutes_to_complete: 10 -author: - - Nii Osae Osae Dade - - Tony Morri - - Sayandip Pal multi_install: false multitool_install_part: false -official_docs: https://github.com/Mindbeam-AI/Litespark-Inference + test_images: - ubuntu:latest test_link: null test_maintenance: true -title: Litespark-Inference -description: Install Litespark-Inference on Arm or x86 Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU. + tool_install: true weight: 1 +layout: installtoolsall --- [Litespark-Inference](https://github.com/Mindbeam-AI/Litespark-Inference) is an open-source CPU inference runtime for [BitNet b1.58](https://arxiv.org/abs/2402.17764) ternary-weight LLMs. A single `pip install` reads your CPU's feature flags and compiles the -right C++ kernel for it - NEON + SDOT on Arm, AVX-512/VNNI or AVX2+FMA -on x86. You do not pick the kernel; it picks itself. +right C++ kernel for it using NEON and SDOT on Arm or AVX-512/VNNI and AVX2+FMA +on x86. This saves you from needing to pick the right C++ kernel for best performance. ## What do I need before installing Litespark-Inference? -- **Python 3.10 or newer.** Check with `python3 --version`. -- **A C++ toolchain** (`clang` or `g++`). Already present on most - developer machines; install steps per platform are below. -- **About 5 GB of free disk.** The BitNet-2B model downloads from - Hugging Face on first run. +- Python 3.10 or newer, you can run `python3 --version` to check your version +- A C++ toolchain such as `clang` or `g++` +- About 5 GB of free disk + +The BitNet-2B model is downloaded from Hugging Face on your first run. It is good practice to install into a clean virtual environment so the install does not conflict with anything else on your machine: -```console +If you don't have Python virtual environment support installed run: + +```bash +sudo apt update +sudo apt install python3-venv -y +``` + +Create a Python virtual environment: + +```bash python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip wheel setuptools ``` -Then follow the section for your platform. +Next, follow the section for your platform, Linux or macOS. -## How do I install Litespark-Inference on Arm or x86 Linux? +## How do I install Litespark-Inference on Arm Linux? The released package builds the correct kernel for your CPU -automatically - AVX-512 with an AVX2+FMA fallback on x86, and NEON + +automatically. It uses NEON and SDOT on Arm (Graviton 2/3/4, Ampere, Neoverse N1/N2/V1/V2, Raspberry -Pi 5). Install the C++ toolchain, then the package: +Pi 5). + +For Ubuntu/Debian distributions, install the C++ toolchain, then the Litespark-Inference Python package: ```bash sudo apt-get update @@ -72,6 +91,7 @@ For Red Hat, Fedora, or RHEL, install the toolchain with `dnf` instead: ```console sudo dnf install -y gcc-c++ clang ninja-build git python3-pip +pip install litespark-inference ``` Confirm the package imports and reports the kernel selected for your CPU: @@ -80,35 +100,42 @@ Confirm the package imports and reports the kernel selected for your CPU: python3 -c "import litespark_inference; print(litespark_inference.__version__)" ``` +The version is printed: + +```output +1.0.3 +``` + To inspect which kernel was built, run: -```console +```bash python -m litespark_inference.torchless info ``` The output ends with one of the following, depending on your CPU: ```output -kernel : avx512 (torchless, extern "C" AVX-512/VNNI + OMP) -kernel : avx2 (torchless, extern "C" AVX2+FMA fallback + OMP) -kernel : neon (torchless, extern "C" NEON SDOT) +litespark_inference.torchless + platform : Linux aarch64 + python : 3.12.3 + kernel : /home/ubuntu/.venv/lib/python3.12/site-packages/litespark_inference/torchless/_matmul_lut_neon.cpython-312-aarch64-linux-gnu.so + OpenMP : True (max_threads=8) + Accelerate: False ``` ## How do I install Litespark-Inference on Apple silicon macOS? Apple's CPUs have NEON SDOT and Litespark-Inference uses it directly. -The only extra step versus Linux is installing `libomp` - Apple's + +The only extra step versus Linux is installing `libomp`, Apple's toolchain does not ship a built-in OpenMP runtime, and Litespark uses -OpenMP for multi-threading inside the kernel: +OpenMP for multi-threading inside the kernel. + +Install Xcode command-line tools, the full Xcode is not required: ```console -# Xcode command-line tools (one-time setup; no full Xcode required) xcode-select --install - -# libomp via Homebrew brew install libomp - -# Install the released package from PyPI pip install litespark-inference ``` @@ -118,18 +145,22 @@ Verify the install: python -m litespark_inference.torchless info ``` -Expected output includes: +The expected output includes: ```output -kernel : neon (torchless, extern "C" NEON SDOT) -OpenMP : True (max_threads=...) +litespark_inference.torchless + platform : Darwin arm64 + python : 3.14.5 + kernel : .venv/lib/python3.14/site-packages/litespark_inference/torchless/_matmul_lut_neon.cpython-314-darwin.so + OpenMP : True (max_threads=12) + Accelerate: True ``` -If `OpenMP : False`, the build did not find Homebrew's `libomp`. The +If you see `OpenMP : False`, the build did not find Homebrew's `libomp`. The most common cause is that Homebrew is installed under `/opt/homebrew` -(the Apple silicon default) but `pip install` ran in an environment that -hides it. Re-running `pip install litespark-inference` from a normal -shell usually fixes it. +(the Apple Silicon default) but `pip install` ran in an environment that +hides it. Re-run `pip install litespark-inference` from a normal +shell to fix it. ## How do I install from source? @@ -144,14 +175,31 @@ pip install -e . ## Sanity-check -On all platforms, the same one-liner generates text. The first run -downloads `microsoft/bitnet-b1.58-2B-4T-bf16` from Hugging Face (around -4.5 GB into `~/.cache/huggingface/hub`); later runs start instantly: +On all platforms, the same command can be used. + +The first run downloads the model weights (around 4.5 GB into +`~/.cache/huggingface/hub`). Subsequent runs do not need to download +again. ```console litespark-inference generate "Hello, world!" --max-tokens 16 ``` -You are now ready to run BitNet-2B. Continue with the -[Accelerate LLM inference on Arm CPUs with Litespark-Inference](/learning-paths/laptops-and-desktops/litespark-inference/) -Learning Path. +The output is similar to: + +```output +Prompt (36 tokens): 'System: You are Litespark, a helpful AI assistant running locally. Provide accurate, concise, and practical answers.<|eot_id|>User: Hello, world!<|eot_id|>Assistant: ' +Prefill... + 36 tokens in 0.40s (90.09 tok/s) +Generate: +Hello! How can I assist you today? + +Generated 9 tokens in 0.29s (31.58 tok/s) + +--- output --- +Hello! How can I assist you today? +``` + +You are now ready to run BitNet-2B. + +Continue with [Accelerate LLM inference on Arm CPUs with Litespark-Inference](/learning-paths/laptops-and-desktops/litespark-inference/) to learn more.