Skip to content

Commit bb4c6d6

Browse files
feat: LFM2.5 text-embedding & ColBERT (MLX/XNNPACK) with prompts and multi-vector output (#1269)
## Description Adds two LFM2.5 retrieval models from Liquid AI and the API needed to use them, through the **existing** `useTextEmbeddings` hook — one native runner, one hook, no new public surface beyond optional model-config fields: - **LFM2.5-Embedding-350M** — dense bi-encoder (CLS pooling, dim 1024). Trained with asymmetric `query: `/`document: ` prompts. - **LFM2.5-ColBERT-350M** — late-interaction retriever (`Linear(1024→128)` per token). Trained with `[Q] `/`[D] ` prompts. Both run on MLX on iOS (physical device) and XNNPACK on Android, quantized (MLX int4, XNNPACK 8da4w). To support them without breaking the existing API, the model config grew three optional fields and `forward` became config-driven: - `prompts` — when present, `forward` requires a `role` (`'query' | 'document'`) and auto-prepends the matching prompt. - `multiVector` — when `true`, `forward` returns a per-token `EmbeddingResult` (`vectors`, `numTokens`, `embeddingDim`, `tokenIds`); otherwise it returns a single pooled `Float32Array` as before. - `skipListIds` — punctuation token ids the consumer excludes from MaxSim scoring. The library auto-applies the role prompts (the matching `query: `/`[Q] ` prefix is prepended in `forward`), but late-interaction scoring (MaxSim) stays the consumer's concern — it runs wherever the vectors are stored. The example app demonstrates one way to score (its own local `maxSim`), and the ColBERT demo is folded into the unified text-embeddings screen, picking the scorer from the model's config. Native side: `TextEmbeddings::generate` returns the raw `[numTokens, embeddingDim]` matrix as an `EmbeddingResult`; the TS layer reduces it. The empty `BaseEmbeddings` base class was removed (`TextEmbeddings` now extends `BaseModel` directly), and output-shape validation was extracted into `TextEmbeddings::buildResult`. **Review order:** start with the TS types (`types/textEmbeddings.ts` — `ForwardFn`/`ForwardReturn` discriminated on the model config), then the module/hook (`TextEmbeddingsModule.ts`, `useTextEmbeddings.ts`), then the native `TextEmbeddings.cpp`/`Types.h`, then the registry/URLs and the example screen. ### Introduces a breaking change? - [ ] Yes - [x] No `forward` stays non-breaking: pooled models still return `Float32Array`. The new return type and `role` requirement only apply to models that opt in via config. ### Type of change - [x] Bug fix (latent: existing models now add CLS/SEP special tokens — see Additional notes) - [x] New feature (change which adds functionality) - [x] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions 1. Open the `text-embeddings` example app. 2. Pick **LFM2.5 Embedding** (MLX on a physical iOS device, XNNPACK on Android/simulator) and run the example queries — weather → "sunny", match → home-team sentences should rank top. 3. Pick **LFM2.5 ColBERT (late-interaction)** — same corpus, scored with MaxSim; ordering should match. 4. Existing pooled models (MiniLM, MPNet, …) keep working unchanged. C++ unit tests: `TextEmbeddingsTests` (incl. new `EmbeddingResult` metadata / `tokenIds` assertions) compiles and links under the Android NDK toolchain. The suite is cross-compiled, so it is not executed on the host in this setup. ### Related issues <!-- Link related issues here using #issue-number --> ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes MLX requires a physical iOS device — the MLX delegate does not run on the simulator (use XNNPACK there). The two models are hosted on the Software Mansion Hugging Face org; docs are updated for both `next` and the `0.9.x` versioned set. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent fb210c3 commit bb4c6d6

38 files changed

Lines changed: 1082 additions & 463 deletions

File tree

.cspell-wordlist.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -215,3 +215,4 @@ MATEUSZ
215215
BLAZEFACE
216216
Blazeface
217217
blazeface
218+
nums

apps/text-embeddings/app/clip-embeddings/index.tsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import {
1717
useTextEmbeddings,
1818
useImageEmbeddings,
1919
ImageEmbeddingsProps,
20+
dotProduct,
2021
} from 'react-native-executorch';
2122

2223
type ImageEmbeddingModel = ImageEmbeddingsProps['model'];
@@ -35,7 +36,6 @@ const IMAGE_MODELS: { label: string; value: ImageEmbeddingModel }[] = [
3536
];
3637
import { launchImageLibrary } from 'react-native-image-picker';
3738
import { useIsFocused } from 'expo-router';
38-
import { dotProduct } from '../../utils/math';
3939
import { ModelPicker } from '../../components/ModelPicker';
4040

4141
const DEFAULT_LABELS = [

0 commit comments

Comments
 (0)