A single C++/ggml binary for retrieval and document understanding — no Python runtime, no ONNX. Text/image/face embeddings, sparse & multi-vector retrieval, rerankers, a full OCR stack (general, scene-text, math, music), layout analysis, NER/KIE, and document preprocessing — all auto-detected from GGUF metadata and GPU-accelerated (CUDA / Vulkan / Metal), with Python, Rust, Dart, HTTP, and WebAssembly front-ends.
Where llama.cpp focuses on text generation, CrispEmbed covers the retrieval, understanding, and document-processing half of the ggml world. 9.5× faster than FastEmbed (ONNX) on MiniLM-L6; runs on Linux, macOS, Windows, iOS, Android, and in the browser.
Live demos: WASM OCR (client-side) · HuggingFace Space (embeddings + math OCR)
| Domain | What it does | Highlights |
|---|---|---|
| Text embeddings | Dense vectors from 10 encoder/decoder architectures | BERT, XLM-R, MPNet, NomicBERT (+MoE), ModernBERT, GTE-v1.5, DeBERTa-v2, Qwen3, Gemma3. Matryoshka truncation, prompt prefixes. cos ≥ 0.965 vs HF |
| Retrieval | Sparse + multi-vector + reranking | SPLADE / BGE-M3 sparse term weights, ColBERT per-token + MaxSim, cross-encoder & bi-encoder rerankers |
| OCR | 15+ engines, image → text/LaTeX/notation | General (DBNet+TrOCR), scene-text (PARSeq), 7 math engines, 4 music (OMR) engines, 6 document VLMs, 12-language Tesseract-LSTM |
| Document AI | Understand page structure | RT-DETRv2 layout (17 classes), Surya text detection, LiLT layout-aware KIE, hOCR/ALTO/searchable-PDF output |
| NER / KIE / LID | Extract structured info | Zero-shot (GLiNER) + fixed-label (BERT/XLM-R) NER, receipt/form KIE, CLD3/GlotLID language ID |
| Vision & face | Cross-modal + biometrics | CLIP/SigLIP text-image search, YuNet/SCRFD detect, ArcFace/SFace/AuraFace recognize |
| Preprocessing | Clean & upscale before OCR | Classical deskew/binarize/dewarp, NAFNet denoise, TPS dewarp, 8 super-resolution engines, PDF-DPI auto-tuning |
Everything ships in one library with a unified C ABI. Over 100 models (200+
GGUF variants) are in the auto-download registry — run crispembed --list-models
for the authoritative, always-current list with per-model license tags.
# Clone (with the ggml submodule) and build
git clone --recursive https://github.com/CrispStrobe/CrispEmbed
cd CrispEmbed
cmake -S . -B build && cmake --build build -j # macOS: ./build-macos.sh (Metal)
# Text embedding (auto-downloads the model by name, or pass a local .gguf)
./build/crispembed -m all-MiniLM-L6-v2 "Hello world"
./build/crispembed -m model.gguf -d 128 "Hello world" # Matryoshka: 128 dims
./build/crispembed -m model.gguf --prefix "query: " "Hello" # prompt prefix
# Retrieval modalities (BGE-M3)
./build/crispembed -m bge-m3 --sparse "Hello world"
./build/crispembed -m bge-m3 --colbert "Hello world"
./build/crispembed -m bge-reranker-v2-m3 --rerank "capital of france" \
"Paris is the capital of France." "Bicycles have two wheels."
# OCR / document AI (engine auto-detected from GGUF metadata)
./build/crispembed -m ppformulanet-l --ocr formula.png # math → LaTeX
./build/crispembed -m flova --ocr score.png # music → LilyPond
./build/crispembed -m transcoda --ocr page.png # full-page score → **kern
./build/crispembed -m qwen3vl-2b --ocr document.png # VLM document OCR
# Cross-modal & face
./build/crispembed -m clip-vit-base-patch16 --image photo.jpg
./build/crispembed -m yunet --detect photo.jpg --json
# HTTP server (text + vision + face + CLIP + OCR + NER in one process)
./build/crispembed-server -m all-MiniLM-L6-v2 --ocr ppformulanet-l-q8_0.gguf --port 8080
curl -X POST http://localhost:8080/embed -d '{"texts": ["Hello world"]}'# Linux / macOS — CPU
cmake -S . -B build && cmake --build build -j
# GPU backends
cmake -S . -B build -DGGML_CUDA=ON && cmake --build build -j # NVIDIA
cmake -S . -B build -DGGML_VULKAN=ON && cmake --build build -j # cross-platform
cmake -S . -B build -DGGML_BLAS=ON && cmake --build build -j # OpenBLAS / MKL
# macOS (recommended: Metal + Accelerate + embedded shaders)
./build-macos.sh # add --cpu for CPU-only, --shared for the Python lib
# Windows (VS 2022 Build Tools + Ninja)
build-windows.bat # or build-vulkan.bat / build-cuda.batRequirements: C++17 compiler, CMake ≥ 3.14. Optional: OpenBLAS, Intel MKL,
CUDA Toolkit, or Vulkan SDK. If you see "ggml does not contain a CMakeLists.txt",
run git submodule update --init --recursive.
./build-ios.sh # CrispEmbed.xcframework (Metal GPU)
./build-android.sh # arm64-v8a + armeabi-v7a + x86_64 (Vulkan/NEON)
./build-wasm.sh # client-side OCR (SIMD / multithreaded / WebGPU tiers)The WASM build runs the full DBNet+TrOCR pipeline, scan-cleanup, and every
auto-detected single-model OCR engine — math → LaTeX, scene text, and music
(OMR: SMT / TrOMR / Flova / Transcoda) — entirely client-side (no server, no API key). Three
tiers: SIMD CPU, multithreaded (COOP/COEP service worker, works on GitHub Pages),
and experimental WebGPU (~2.8× on ViT recognition, ~60× on DBNet detection vs
WASM CPU). The whole engine set is one 2.3 MB .wasm. See examples/wasm-ocr/README.md.
cmake --install build --prefix /usr/local lays out a standard tree with a
versioned .so/.dylib (SONAME, RPATH=$ORIGIN), CMake package config, and a
relocatable pkg-config file:
find_package(crispembed REQUIRED)
target_link_libraries(my_app PRIVATE crispembed::crispembed)Ten architectures, auto-detected from GGUF tensor names. Dense, sparse, and multi-vector heads all run through ggml graphs with GPU dispatch.
28 embedding models validated at cos ≥ 0.965; a representative slice:
| Model | Type | Dim | F32 | Q8_0 | Q4_K |
|---|---|---|---|---|---|
| all-MiniLM-L6-v2 | BERT | 384 | 0.999999 | 0.9995 | 0.97 |
| multilingual-e5-large | XLM-R | 1024 | 0.999997 | 0.9999 | 0.99 |
| gte-modernbert-base | ModernBERT | 768 | 0.999991 | 0.9999 | — |
| nomic-embed-text-v2-moe | NomicBERT MoE | 768 | 1.000000 | 0.9996 | 0.966 |
| EmbeddingGemma-300m | Gemma3 | 768 | 1.000000 | 0.9998 | 0.98 |
| Qwen3-Embedding-0.6B | Qwen3 | 1024 | 0.999895 | 0.9996 | 0.97 |
| Octen-Embedding-8B | Qwen3 | 4096 | — | — | 0.965 |
Q8_0 = all PASS (cos ≥ 0.995). — in Q4_K = SwiGLU/GeGLU too sensitive for
aggressive quants (defaults to Q8_0). The full table lives in
PERFORMANCE.md.
CrispEmbed also loads the official/community gemma-embedding GGUFs directly
(llama.cpp SPM exports, e.g. ggml-org/embeddinggemma-300m-*-GGUF). These ship
without the SentenceTransformers Dense head — llama.cpp applies it from an
external file — so their raw output is the backbone mean-pool. Bake the Dense
head in with models/add-st-dense-to-gguf.py for HF-compatible embeddings
(cos 0.984 vs HF), or just pull the ready-made embeddinggemma-300m-qat.
from crispembed import CrispEmbed
model = CrispEmbed("bge-m3.gguf")
vec = model.encode("Hello world") # dense, L2-normalized (1024,)
sparse = model.encode_sparse("Hello world") # {token_id: weight} (SPLADE-style)
multi = model.encode_multivec("Hello world") # (n_tokens, 128) (ColBERT)
reranker = CrispEmbed("bge-reranker-v2-m3.gguf")
score = reranker.rerank("query", "document") # cross-encoder logit
ranked = model.rerank_biencoder("query", ["d1","d2"], top_n=2) # cosineLFM2.5-ColBERT (128-d per token) and all seven cross-encoder rerankers are
supported. Sparse/ColBERT heads are written into the GGUF by the converter and
detected via has_sparse / has_colbert.
15+ engines for image → text, most auto-detected from GGUF metadata via the
unified crispembed_ocr_model_* C API. Available through CLI (--ocr), server
(POST /ocr/model), Python (CrispOcrModel), Rust, and Dart/Flutter.
| Model | Architecture | Params | Use case | License |
|---|---|---|---|---|
| PARSeq | ViT + Transformer | 24M | Scene text (SOTA, ECCV'22) | Apache-2.0 |
| DBNet + TrOCR | ResNet-18+FPNC → DeiT+Transformer | 7+63M | General doc pipeline (~200ms/region) | MIT / Apache-2.0 |
| Tesseract-LSTM | VGSL Conv+LSTM+CTC | <2 MB | 12 languages, tiny GGUFs | Apache-2.0 |
| PP-FormulaNet-L | SAM-ViT + MBart | 181M | Printed math (best) | Apache-2.0 |
| MixTeX | Swin-Tiny + RoBERTa | 86M | CN+EN LaTeX | Apache-2.0 |
| Texo-Distill | HGNetv2 + MBart | 20M | Printed math (small) | AGPL-3.0 |
| PosFormer / BTTR / HMER | DenseNet + Transformer/GRU | 6–7M | Handwritten math (CROHME) | MIT / CC-BY-NC |
| SMT | ConvNext + Transformer | 21M | Printed music (systems) → bekern (96.3% GrandStaff) | MIT |
| SMT++ full-page | ConvNext + Transformer | 11M | Whole pianoform page → bekern (no segmentation) | MIT |
| Polyphonic-TrOMR | ResNetV2+ViT + 4-head decoder | ~22M | Printed music photos → symbolic | Apache-2.0 |
| Flova/omr_transformer | DonutSwin + mBART-4L | 143M | Handwritten/whiteboard music → LilyPond | Apache-2.0 |
| Transcoda-59M | ConvNeXt-V2 + 8L RoPE cross-attn | 59M | Zero-shot full-page score → Humdrum **kern (real-scan SOTA) |
CC-BY-4.0 |
| GOT-OCR2 | SAM ViT-B + Qwen2-0.5B | 0.7B | Doc OCR (text+LaTeX+tables) | Apache-2.0 |
| GLM-OCR | CogViT + GLM-0.5B | 0.9B | Doc OCR (OmniDocBench #1, 8 langs) | MIT |
| InternVL2 / 2.5 | InternViT + Qwen2/InternLM2.5 | 0.9–2.1B | Edge/WASM & EN+DE VLM OCR | MIT |
| Qwen2.5-VL / Qwen3-VL | ViT (+DeepStack) + Qwen LLM | 2.4–3.6B | General/multilingual VLM OCR | Apache-2.0 |
| DeepSeek-OCR-2 / Unlimited-OCR | dual ViT + DeepSeek-V2 MoE | 3–3.3B | Full-page doc OCR + layout grounding | Apache-2.0 / MIT |
| Qari-OCR | Qwen2-VL-2B + LoRA | 2B | Arabic OCR with diacritics | Apache-2.0 |
Formula/music engines validated per-stage against their HF references (typically
cos ≥ 0.999, byte-exact greedy decode). VLM engines ingest the full page and
letterbox internally — the pipeline skips scan-cleanup for them.
CRISPEMBED_MAX_PIXELS trades resolution for CPU speed on all variable-resolution
VLMs.
Three permissively-licensed engines, all auto-detected via --ocr:
- SMT (MIT,
smt-grandstaff) — printed polyphonic staff systems → bekern. Reproduces the reference exactly (per-stage cos = 1.0, 96.3% vs GrandStaff). - SMT++ full-page (MIT,
smt-fp) — a whole pianoform page → bekern in one pass, no staff/system segmentation (per-stage cos ≥ 0.9998, byte-exact greedy decode vs the HF reference). - Polyphonic-TrOMR (Apache-2.0) — staff photos → rhythm/pitch/lift streams. Robust on real photos; byte-exact decode on the reference examples.
- Flova/omr_transformer (Apache-2.0) — the only permissive handwritten
music model; whiteboard "simple notes" → LilyPond, byte-exact incl. the native
no-
transformerspreprocessing path. - Transcoda-59M (CC-BY-4.0,
transcoda) — zero-shot full-page score → Humdrum**kernin one pass. ConvNeXt-V2-Tiny encoder + 8-layer RoPE cross-attention decoder; OMR-NED SOTA on real historical scans. Clean-room engine (per-stage cos = 1.0, byte-exact greedy decode vs the HF reference).
- Layout detection — RT-DETRv2 (ResNet-50 + deformable decoder), 17 region
types.
--layout,POST /layout/detect. Encoder cos = 1.0 vs HF; Q8_0 43 MB. - Text detection — Surya EfficientViT segformer (38M, 91 languages,
GPU-accelerated), plus a model-free connected-component fallback (0 downloads,
4 ms/page).
--text-detect. - Scan cleanup — Tier 1 classical (deskew with dual-detector consensus,
Otsu/Sauvola binarize, border crop, background whitening, cubic-baseline
dewarp, 1-bit DWA morphology — 21× faster than float, all reimplemented from
Leptonica). Tier 2 learned NAFNet denoise.
--cleanup-only/--cleanup. - Text super-resolution — PAN (4× whole-page, 0.5 MB), TBSRN (2× per-line, 2 MB), NAFNet-SR scaffold; parity cos ≥ 0.9996. Eight SR backbones total (HAT, DAT, ESRGAN, SwinIR, TBSRN, SAFMN, Restormer, SCUNet).
- PDF DPI profiling — zero-dependency PDF parser computes effective page DPI to auto-select OCR resolution (downsample high-DPI, super-resolve low-DPI).
- Output formats — plain text, hOCR, ALTO 3.1 XML, searchable PDF, with multi-page accumulation. An orchestrator routes by source type (screenshot/scan/photo) with accept-gate cascading and VLM fallback.
- NER — zero-shot GLiNER (LFM2.5-350M bidirectional backbone; arbitrary
entity types at inference, all 16 layers cos = 1.0 vs HF) and fixed-label
BERT/XLM-R (
bert-base-nerEN,xlmr-ner-hrl10 languages). One--nerAPI, backend auto-detected. - KIE — chains OCR + GLiNER to pull key-value fields from receipts/invoices/
forms, no new model.
--kie,POST /kie/extract,CrispKIE. - LiLT — layout-aware document understanding (RoBERTa + layout transformer via BiACM), 130M, MIT, FUNSD token classification. 25/25 layers cos = 1.0.
- LID — CLD3 (109 langs) / GlotLID (2102 ISO 639-3) text language ID, used to auto-select the Tesseract model in the OCR pipeline.
./build/crispembed -m gliner-lfm --ner "Maria Schmidt arbeitet bei Siemens in München"
# Maria Schmidt → person, Siemens → organization, München → locationCLIP and SigLIP text-image cross-modal search (shared vector space), plus a full face pipeline: YuNet (0.2 MB) / SCRFD (16 MB) detection → ArcFace / SFace / AuraFace recognition.
./build/crispembed -m clip-text-base "a photo of a cat"
./build/crispembed -m clip-vit-base-patch16 --image photo.jpg
./build/crispembed -m yunet --detect photo.jpg --jsonBidirLM-Omni unifies text, audio, and image into one shared 2048-d space (bidirectional Qwen3 body + Whisper-shape audio encoder + Qwen2VL vision tower with DeepStack). Q4_K verified locally across all three modalities.
Python, Rust, Dart, and the CLI expose the same core inference features from the shared C ABI (dense/batch encode, Matryoshka, prefix, sparse, ColBERT, rerank).
# Python (needs the shared lib: --shared or -DCRISPEMBED_BUILD_SHARED=ON)
from crispembed import CrispEmbed
model = CrispEmbed("all-MiniLM-L6-v2.gguf")
vecs = model.encode(["Hello world", "Goodbye world"]) # (2, 384), one batched GPU call
model.set_dim(128); model.set_prefix("query: ")// Rust — crispembed = { git = "https://github.com/CrispStrobe/CrispEmbed" }
let mut model = crispembed::CrispEmbed::new("model.gguf", 0)?;
let vec = model.encode("Hello world");// Dart / Flutter (iOS Metal, Android Vulkan/NEON, desktop)
final model = CrispEmbed('model.gguf');
final vec = model.encode('Hello world'); // Float32List(384)/* C ABI */
void *ctx = crispembed_ocr_model_init("ppformulanet-l-q8_0.gguf", 4);
const char *latex = crispembed_ocr_model_recognize(ctx, pixels, w, h, ch, &len);Per-language parity scripts (tests/feature_parity.py, the Rust/Dart
feature_parity examples) verify the wrappers against the CLI. All 45+ registry
models also export as Ollama-compatible GGUFs (--ollama converter flag).
# Encoders (BERT / XLM-R) and decoders (Qwen3 / Gemma3)
pip install torch transformers gguf
python models/convert-bert-to-gguf.py --model sentence-transformers/all-MiniLM-L6-v2 --output out.gguf --crisp
python models/convert-decoder-embed-to-gguf.py --model Octen/Octen-Embedding-0.6B --output octen.gguf
# Quantize (Q8_0 recommended; Q4_K for max compression)
./build/crispembed-quantize model.gguf model-q8_0.gguf q8_0
# Import a stock llama.cpp VL model (LLM GGUF + mmproj) byte-for-byte, no re-quant
python models/merge-llamacpp-gguf.py --llm InternVL2_5-1B-Q8_0.gguf \
--mmproj mmproj-InternVL2_5-1B-f16.gguf --output internvl2_5-1b-crispembed.ggufGGUFs are quantized with an importance matrix (imatrix, activation-weighted),
A/B-validated per model class with a task-appropriate metric (mean cosine for
embedders, Kendall-τ for rerankers, span-F1 for NER, etc.). -m <model>
auto-downloads each model's best-tested small flavor; -q8 / -q4k / -iq4xs
suffixes pick a specific variant.
| Type | Compression | Quality (cos vs F32) |
|---|---|---|
| Q8_0 | ~3.8× | > 0.995 (recommended) |
| Q6_K | ~4.5× | > 0.99 |
| Q5_K | ~5× | > 0.98 |
| Q4_K | ~5.5× | > 0.95 (max compression) |
Pre-converted models: huggingface.co/cstr.
Apple M1, Metal, all-MiniLM-L6-v2:
| Engine | Single text | Batch (10) |
|---|---|---|
| CrispEmbed (Python ctypes) | 3.6 ms / 280 t/s | 12.7 ms / 787 t/s |
| fastembed-rs (Rust ONNX) | 3.8 ms / 263 t/s | 18.9 ms / 528 t/s |
| HuggingFace (PyTorch) | 12.2 ms / 82 t/s | 29.8 ms / 335 t/s |
Full multi-model and Ollama Q8_0/Q4_K numbers in PERFORMANCE.md.
Benchmark with ./benchmark.sh [--multi].
Part of the Crisp ecosystem, and complementary to llama.cpp (shared ggml backend, different problem space — retrieval/understanding vs generation):
| Project | Role |
|---|---|
| CrispEmbed | This repo — embedding + retrieval + document-AI engine (ggml) |
| CrispASR | Speech recognition (11 ASR backends) + text NMT; shares the ggml core |
| crisp-docx | .docx surgery + document translation; uses CrispEmbed word alignment |
| CrispSorter | Tauri desktop organiser; LanceDB indexer on CrispEmbed embeddings |
Capabilities llama.cpp does not cover: sparse/ColBERT retrieval, cross-encoder reranking, the OCR/layout/detection/cleanup/super-resolution document stack, face detect+recognize, NER/KIE, and a client-side WASM OCR build.
Model type is auto-detected from GGUF metadata at load:
- Encoders (BERT/XLM-R/MPNet/NomicBERT/ModernBERT/GTE-v1.5/DeBERTa-v2/SPLADE)
→
src/crispembed.cpp. Variants detected from tensor names (RoPE vs learned positions, rel-attn-bias, pre-LN, fused GeGLU). - Decoders (Qwen3/Gemma3/BidirLM-Omni) →
src/decoder_embed.cpp. - Vision / audio (BidirLM-Omni) →
src/bidirlm_vision.cpp/src/bidirlm_audio.cpp, opened lazily.
The HTTP server exposes four embedding dialects (native, OpenAI, Ollama batch & legacy) plus face, ViT/CLIP, OCR, NER, LID, KIE, document-OCR, and preprocessing endpoints. See PLAN.md (roadmap), HISTORY.md (milestones), and LEARNINGS.md (deep dives) for detail.
Converting a checkpoint to GGUF does not relicense it — each downloaded model
is governed by its upstream license. Check the License column in
--list-models (or the upstream model card) before commercial use.
| License class | Examples | What you can do |
|---|---|---|
| Permissive (Apache-2.0 / MIT / CC-BY-4.0) | most BERT/XLM-R/MPNet, BGE, E5, Granite, MXBai, Nomic, Qwen3, Harrier, GTE-v1.5, SMT, TrOMR, Flova, Transcoda (CC-BY-4.0), LiLT | commercial use OK with normal attribution |
| CC BY-NC 4.0 (non-commercial) | jina-v5-*, jina-reranker-v2, PosFormer (ours) |
research/eval only; commercial needs a vendor license |
| LFM Open License v1.0 | lfm2-embed*, lfm2-colbert, gliner-lfm |
free under $10M annual revenue |
| Gemma Terms | embeddinggemma-300m |
commercial OK, subject to Google's Prohibited Use Policy |
| Other restricted | Surya weights (OpenRAIL-M, free < $5M), Texo (AGPL) | see the model card |
Restricted entries are flagged with * in --list-models and require explicit
consent to auto-download (interactive prompt, --accept-license <spdx>, or
CRISPEMBED_ACCEPT_LICENSE). --accept-license acknowledges the caller accepts
upstream terms — it does not grant rights you don't otherwise have. Audit the
whole registry with python tests/check_registry_licenses.py.
CrispEmbed's own code is MIT (see LICENSE), consistent with its
ggml/llama.cpp foundation.
Per-model weights are covered by their respective upstream/HuggingFace model
licenses (see Model licenses and --list-models) — converting
a checkpoint to GGUF does not change its license. The crispembed binary itself
links model runtimes that are mostly permissively licensed (MIT / Apache-2.0 /
CC-BY-4.0 for weights); a few registry models carry non-commercial or
vendor-specific terms and are flagged accordingly.
- ggml — inference engine
- CrispASR — shared core (gguf_loader, bpe, crisp_audio)
- sentence-transformers — ground-truth validation
- The upstream model authors — see each model's card for architecture credit