feat(embedder)!: swap embedding model gte-modernbert-base → F2LLM-v2-80M (320-dim)

[theagenticguy]

Summary

Swaps the local ONNX embedder from gte-modernbert-base (768-dim, CLS pooling, JS-side L2 norm) to codefuse-ai/F2LLM-v2-80M (320-dim, last-token pooling + L2 norm baked into the ONNX graph, Apache-2.0). Adds a query/document asymmetry: queries get F2LLM's Instruct: prefix; documents embed raw.

Warning

BREAKING — requires re-index. 320-dim query vectors can't be compared against stored 768-dim vectors. Existing indexes must be rebuilt with codehub analyze --embeddings. The embedder-fingerprint guard (ADR 0014) already refuses queries against a stale store, and the analyze path now suppresses the content-hash cache on a model-id change so a swap forces a full re-embed (never a mixed-dimension store).

What changed

Embedder core (packages/embedder)

• onnx-embedder.ts requests the in-graph embedding output (shape [B,320]) instead of last_hidden_state; deleted clsPool + l2NormalizeInPlace (the graph pools + normalizes). Qwen2 pad id (<|endoftext|>=151643).
• New Embedder.embedQuery() + query-prefix.ts apply Instruct: {instruction}\nQuery: {q} to queries only. Wired at the search/hybrid.ts query seam; documents stay raw.
• model-pins.ts: GTE_MODERNBERT_BASE_PINS → F2LLM_V2_80M_PINS. Weights are a custom ONNX export (not on HF) hosted as GitHub release embed-v1, SHA256-pinned. 3-file manifest (model + tokenizer.json + tokenizer_config.json); config.json/special_tokens_map.json dropped.
• Package bumped 0.1.3 → 0.2.0.

Dimension 768 → 320 across embedder, search (NullEmbedder/DEFAULT_EMBEDDER_DIM), storage (DEFAULT_DIM), ingestion (embedder-pool), and HTTP/SageMaker defaults.

Migration safety — cli/commands/analyze.ts openEmbeddingHashCacheAdapter(repoPath, activeModelId) suppresses the content-hash cache when the prior store's embedderModelId differs.

Docs / housekeeping — swept README, SPECS, package docs, CHANGELOGs, skills to F2LLM/320-dim. Collapsed the stale storage ADR chain (0001/0011/0013-m7/0016 now all point forward to ADR 0019; 0016 no longer falsely reads "Accepted") and fixed the README's pre-ADR-0019 storage narrative (it still described native lbug+DuckDB bindings and a platform-prebuild matrix that no longer exist). Fixed the lefthook verdict guard to check store.sqlite (was the removed graph.lbug).

Verification

• pnpm -r build + pnpm -r test green — ~2,400 tests, 0 failures; biome + banned-strings pass; pre-push hooks (typecheck + test + verdict) green.
• Tokenizer parity: JS @huggingface/tokenizers produces byte-identical token IDs to the Python AutoTokenizer the POC used (the one risk the POC never tested — now closed).
• Ranking parity: the production OnnxEmbedder reproduces the POC's 4/4 top-1 ranking and byte-deterministic output (satisfies the graphHash gate).
• End-to-end: analyze --embeddings writes 320-dim rows + f2llm-v2-80m/fp32 stamp; hybrid query ranks correctly; migration guard confirmed firing on a stale gte-modernbert stamp.
• Release verified: embed-v1 assets uploaded; pinned URLs resolve and checksums match; a real codehub setup --embeddings install from the live release passes streaming SHA256 verification.

Attribution

F2LLM — Zhang, Liao, Yu, Di, Wang (arXiv:2603.19223), Apache-2.0.