Commit 462663b
committed
feat: Add Nomic BERT model support
Add support for nomic-ai/nomic-embed-text-v1.5 embedding model.
Architecture:
- Postnorm transformer (standard BERT-style)
- SwiGLU activation (up * silu(gate))
- Rotary position embeddings (RoPE) with base 1000
- Combined Wqkv projection
- No biases in attention and FFN layers
- Mean pooling over non-masked tokens
Tested against Python transformers with ~2e-6 precision.1 parent 2252d44 commit 462663b
4 files changed
Lines changed: 450 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
| 179 | + | |
179 | 180 | | |
180 | 181 | | |
181 | 182 | | |
| |||
266 | 267 | | |
267 | 268 | | |
268 | 269 | | |
| 270 | + | |
269 | 271 | | |
270 | 272 | | |
271 | 273 | | |
| |||
0 commit comments