11---
22title : Litespark-Inference
33
4- description : Install Litespark-Inference on Arm or x86 Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU.
4+ description : Install Litespark-Inference on Arm Linux and Apple silicon macOS to run BitNet ternary LLMs on the CPU.
55
66minutes_to_complete : 10
77
@@ -38,10 +38,10 @@ layout: installtoolsall
3838
3939[ Litespark-Inference] ( https://github.com/Mindbeam-AI/Litespark-Inference )
4040is an open-source CPU inference runtime for
41- [ BitNet b1.58] ( https://arxiv.org/abs/2402.17764 ) ternary-weight LLMs. A
42- single ` pip install ` reads your CPU's feature flags and compiles the
43- right C++ kernel for it using NEON and SDOT on Arm or AVX-512/VNNI and AVX2+FMA
44- on x86 . This saves you from needing to pick the right C++ kernel for best performance.
41+ [ BitNet b1.58] ( https://arxiv.org/abs/2402.17764 ) ternary-weight LLMs.
42+
43+ A single ` pip install ` reads your CPU's feature flags and compiles the
44+ right C++ kernel for it using Neon and SDOT on Arm . This saves you from needing to manually pick the right C++ kernel for best performance.
4545
4646You can install Litespark-Inference on Linux and macOS.
4747
@@ -55,10 +55,10 @@ Before installing Litespark-Inference, make sure your local machine has the foll
5555
5656To check the Python version on your machine, run ` python3 --version ` .
5757
58- The BitNet-2B model is downloaded from Hugging Face on your first run .
58+ The first run downloads BitNet-2B model from Hugging Face.
5959
60- It's a good practice to install into a clean virtual environment so the
61- install doesn't conflict with anything else on your machine:
60+ Install Litespark-Inference into a clean virtual environment so the
61+ install doesn't conflict with anything else on your machine.
6262
6363If you don't have Python virtual environment support installed, run:
6464
@@ -78,13 +78,12 @@ python -m pip install --upgrade pip wheel setuptools
7878## Install Litespark-Inference on Arm Linux
7979
8080The released package builds the correct kernel for your CPU
81- automatically. It uses Neon and
82- SDOT on Arm.
81+ automatically. It uses Neon and SDOT on Arm.
8382
8483Supported CPUs include:
8584
8685- AWS Graviton 2, 3, or 4
87- - Ampere
86+ - Ampere family processors
8887- Neoverse N1, N2, V1, or V2
8988- Raspberry Pi 5
9089
@@ -103,7 +102,7 @@ sudo dnf install -y gcc-c++ clang ninja-build git python3-pip
103102pip install litespark-inference
104103```
105104
106- Confirm the package imports and reports the kernel selected for your CPU :
105+ Confirm the package imports and list the installed version of Litespark-Inference :
107106
108107``` bash
109108python3 -c " import litespark_inference; print(litespark_inference.__version__)"
@@ -115,7 +114,7 @@ The output is similar to:
1151141.0.3
116115```
117116
118- To inspect which kernel was built, run :
117+ Inspect which kernel was built for your CPU :
119118
120119``` bash
121120python -m litespark_inference.torchless info
@@ -139,15 +138,15 @@ Apple's CPUs have Neon SDOT, and Litespark-Inference uses it directly.
139138Litespark uses OpenMP for multi-threading inside the kernel. However, Apple's
140139toolchain doesn't ship a built-in OpenMP runtime. To address this, you'll need to install ` libomp ` .
141140
142- Install Xcode command-line tools, ` libomp ` , and the Litespark-Inference Python package. The full set of Xcode tools isn't required :
141+ Install Xcode command-line tools, ` libomp ` , and the Litespark-Inference Python package:
143142
144143``` console
145144xcode-select --install
146145brew install libomp
147146pip install litespark-inference
148147```
149148
150- Verify the install :
149+ Verify that the installation was successful :
151150
152151``` console
153152python -m litespark_inference.torchless info
@@ -164,15 +163,19 @@ litespark_inference.torchless
164163 Accelerate: True
165164```
166165
166+ ### Troubleshoot missing OpenMP support
167+
167168If you see ` OpenMP : False ` , the build didn't find Homebrew's ` libomp ` . The
168- most common cause is that Homebrew is installed under ` /opt/homebrew `
169- (the Apple silicon default), but ` pip install ` ran in an environment that
170- hides it. Re-run ` pip install litespark-inference ` from a normal
171- shell to fix it.
169+ most common cause is that Homebrew is installed under ` /opt/homebrew ` , which is the
170+ Apple silicon default, but ` pip install ` ran in an environment that
171+ hides it.
172+
173+ To fix this, re-run ` pip install litespark-inference ` from a normal
174+ shell.
172175
173- ## Install Litespark-Inference from source
176+ ## (Optional) Install Litespark-Inference from source
174177
175- To modify the runtime or kernels, install from source instead of from
178+ To modify the runtime or kernels, install from source rather than from
176179PyPI:
177180
178181``` console
0 commit comments