Commit bb4c6d6
feat: LFM2.5 text-embedding & ColBERT (MLX/XNNPACK) with prompts and multi-vector output (#1269)
## Description
Adds two LFM2.5 retrieval models from Liquid AI and the API needed to
use them, through the **existing** `useTextEmbeddings` hook — one native
runner, one hook, no new public surface beyond optional model-config
fields:
- **LFM2.5-Embedding-350M** — dense bi-encoder (CLS pooling, dim 1024).
Trained with asymmetric `query: `/`document: ` prompts.
- **LFM2.5-ColBERT-350M** — late-interaction retriever
(`Linear(1024→128)` per token). Trained with `[Q] `/`[D] ` prompts.
Both run on MLX on iOS (physical device) and XNNPACK on Android,
quantized (MLX int4, XNNPACK 8da4w).
To support them without breaking the existing API, the model config grew
three optional fields and `forward` became config-driven:
- `prompts` — when present, `forward` requires a `role` (`'query' |
'document'`) and auto-prepends the matching prompt.
- `multiVector` — when `true`, `forward` returns a per-token
`EmbeddingResult` (`vectors`, `numTokens`, `embeddingDim`, `tokenIds`);
otherwise it returns a single pooled `Float32Array` as before.
- `skipListIds` — punctuation token ids the consumer excludes from
MaxSim scoring.
The library auto-applies the role prompts (the matching `query: `/`[Q] `
prefix is prepended in `forward`), but late-interaction scoring (MaxSim)
stays the consumer's concern — it runs wherever the vectors are stored.
The example app demonstrates one way to score (its own local `maxSim`),
and the ColBERT demo is folded into the unified text-embeddings screen,
picking the scorer from the model's config.
Native side: `TextEmbeddings::generate` returns the raw `[numTokens,
embeddingDim]` matrix as an `EmbeddingResult`; the TS layer reduces it.
The empty `BaseEmbeddings` base class was removed (`TextEmbeddings` now
extends `BaseModel` directly), and output-shape validation was extracted
into `TextEmbeddings::buildResult`.
**Review order:** start with the TS types (`types/textEmbeddings.ts` —
`ForwardFn`/`ForwardReturn` discriminated on the model config), then the
module/hook (`TextEmbeddingsModule.ts`, `useTextEmbeddings.ts`), then
the native `TextEmbeddings.cpp`/`Types.h`, then the registry/URLs and
the example screen.
### Introduces a breaking change?
- [ ] Yes
- [x] No
`forward` stays non-breaking: pooled models still return `Float32Array`.
The new return type and `role` requirement only apply to models that opt
in via config.
### Type of change
- [x] Bug fix (latent: existing models now add CLS/SEP special tokens —
see Additional notes)
- [x] New feature (change which adds functionality)
- [x] Documentation update (improves or adds clarity to existing
documentation)
- [ ] Other (chores, tests, code style improvements etc.)
### Tested on
- [x] iOS
- [x] Android
### Testing instructions
1. Open the `text-embeddings` example app.
2. Pick **LFM2.5 Embedding** (MLX on a physical iOS device, XNNPACK on
Android/simulator) and run the example queries — weather → "sunny",
match → home-team sentences should rank top.
3. Pick **LFM2.5 ColBERT (late-interaction)** — same corpus, scored with
MaxSim; ordering should match.
4. Existing pooled models (MiniLM, MPNet, …) keep working unchanged.
C++ unit tests: `TextEmbeddingsTests` (incl. new `EmbeddingResult`
metadata / `tokenIds` assertions) compiles and links under the Android
NDK toolchain. The suite is cross-compiled, so it is not executed on the
host in this setup.
### Related issues
<!-- Link related issues here using #issue-number -->
### Checklist
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have updated the documentation accordingly
- [x] My changes generate no new warnings
### Additional notes
MLX requires a physical iOS device — the MLX delegate does not run on
the simulator (use XNNPACK there). The two models are hosted on the
Software Mansion Hugging Face org; docs are updated for both `next` and
the `0.9.x` versioned set.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent fb210c3 commit bb4c6d6
38 files changed
Lines changed: 1082 additions & 463 deletions
File tree
- apps/text-embeddings
- app
- clip-embeddings
- text-embeddings
- utils
- docs
- docs
- 03-hooks/01-natural-language-processing
- 04-typescript-api/01-natural-language-processing
- versioned_docs/version-0.9.x
- 03-hooks/01-natural-language-processing
- 04-typescript-api/01-natural-language-processing
- 06-api-reference
- classes
- functions
- interfaces
- type-aliases
- packages/react-native-executorch
- common/rnexecutorch
- host_objects
- models
- embeddings
- text
- text_to_image
- tests
- integration
- src
- constants
- textEmbeddings
- hooks/natural_language_processing
- modules/natural_language_processing
- types
- utils
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
| 218 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
38 | | - | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
0 commit comments