Skip to content

Commit 462663b

Browse files
feat: Add Nomic BERT model support
Add support for nomic-ai/nomic-embed-text-v1.5 embedding model. Architecture: - Postnorm transformer (standard BERT-style) - SwiGLU activation (up * silu(gate)) - Rotary position embeddings (RoPE) with base 1000 - Combined Wqkv projection - No biases in attention and FFN layers - Mean pooling over non-masked tokens Tested against Python transformers with ~2e-6 precision.
1 parent 2252d44 commit 462663b

4 files changed

Lines changed: 450 additions & 0 deletions

File tree

lib/bumblebee.ex

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,7 @@ defmodule Bumblebee do
176176
"MPNetForTokenClassification" => {Bumblebee.Text.MpNet, :for_token_classification},
177177
"MPNetForQuestionAnswering" => {Bumblebee.Text.MpNet, :for_question_answering},
178178
"MPNetForMultipleChoice" => {Bumblebee.Text.MpNet, :for_multiple_choice},
179+
"NomicBertModel" => {Bumblebee.Text.NomicBert, :base},
179180
"PhiModel" => {Bumblebee.Text.Phi, :base},
180181
"PhiForCausalLM" => {Bumblebee.Text.Phi, :for_causal_language_modeling},
181182
"PhiForSequenceClassification" => {Bumblebee.Text.Phi, :for_sequence_classification},
@@ -266,6 +267,7 @@ defmodule Bumblebee do
266267
"mistral" => :llama,
267268
"mbart" => :mbart,
268269
"mpnet" => :mpnet,
270+
"nomic_bert" => :bert,
269271
"phi" => :code_gen,
270272
"phi3" => :llama,
271273
"qwen3" => :qwen2,

0 commit comments

Comments
 (0)