Summary
RubyLLM.embed returns only the dense vector. Sparse-capable embedding models (BGE-M3, SPLADE) emit a second, sparse representation in the same /v1/embeddings response, which RubyLLM drops — so dense+sparse hybrid retrieval isn't possible through the gem.
This is a de-facto extension, not (yet) in the OpenAI spec
The official OpenAI embeddings response is dense-only — data[].embedding is a plain float array, with no sparse field (API reference). Sparse output is therefore a de-facto extension that OpenAI-compatible servers add on top; how to expose it is still being drafted in the OSS ecosystem (vLLM #13609, #33882). This issue proposes RubyLLM track the form that's converging there.
Proposed: stabilize on a sparse_embedding / lexical_weights dict
Expose an additive RubyLLM::Embedding#sparse_vectors, normalized to a { token_id => Float } map, populated when a /v1/embeddings data[] row carries sparse_embedding or lexical_weights; nil for dense-only providers (fully backward compatible).
Why this form (dict {token_id => weight}), and not the alternatives:
- It's the shape that appears inside the
/v1/embeddings response — the layer this gem parses. BGE-M3 emits lexical_weights as exactly this dict.
- It's where the production side is converging: vLLM's sparse-output redesign (#33882) proposes returning
Dict[int, float]; Elasticsearch sparse_vector / ELSER also consume a token→weight object.
- It's idiomatic Ruby (a
Hash) and trivially convertible to the parallel-array { indices: [...], values: [...] } that vector DBs (Pinecone, Qdrant) want on ingest — but that's a downstream storage format, a different layer, not what an embedding server returns.
- HuggingFace TEI exposes sparse only via a separate
/embed_sparse route (a [{index, value}] list), not on /v1/embeddings — out of scope for the OpenAI provider parse, optionally a thin separate helper later.
Current behavior
lib/ruby_llm/providers/openai/embeddings.rb keeps only d['embedding']; RubyLLM::Embedding exposes only vectors. Any sibling sparse field is discarded.
Backward compatibility
Additive only — #vectors unchanged, new #sparse_vectors is nil for every provider that doesn't return sparse. Only the OpenAI-compatible provider parse changes. Happy to send a PR.
Summary
RubyLLM.embedreturns only the dense vector. Sparse-capable embedding models (BGE-M3, SPLADE) emit a second, sparse representation in the same/v1/embeddingsresponse, which RubyLLM drops — so dense+sparse hybrid retrieval isn't possible through the gem.This is a de-facto extension, not (yet) in the OpenAI spec
The official OpenAI embeddings response is dense-only —
data[].embeddingis a plain float array, with no sparse field (API reference). Sparse output is therefore a de-facto extension that OpenAI-compatible servers add on top; how to expose it is still being drafted in the OSS ecosystem (vLLM #13609, #33882). This issue proposes RubyLLM track the form that's converging there.Proposed: stabilize on a
sparse_embedding/lexical_weightsdictExpose an additive
RubyLLM::Embedding#sparse_vectors, normalized to a{ token_id => Float }map, populated when a/v1/embeddingsdata[]row carriessparse_embeddingorlexical_weights;nilfor dense-only providers (fully backward compatible).Why this form (dict
{token_id => weight}), and not the alternatives:/v1/embeddingsresponse — the layer this gem parses. BGE-M3 emitslexical_weightsas exactly this dict.Dict[int, float]; Elasticsearchsparse_vector/ ELSER also consume a token→weight object.Hash) and trivially convertible to the parallel-array{ indices: [...], values: [...] }that vector DBs (Pinecone, Qdrant) want on ingest — but that's a downstream storage format, a different layer, not what an embedding server returns./embed_sparseroute (a[{index, value}]list), not on/v1/embeddings— out of scope for the OpenAI provider parse, optionally a thin separate helper later.Current behavior
lib/ruby_llm/providers/openai/embeddings.rbkeeps onlyd['embedding'];RubyLLM::Embeddingexposes onlyvectors. Any sibling sparse field is discarded.Backward compatibility
Additive only —
#vectorsunchanged, new#sparse_vectorsisnilfor every provider that doesn't return sparse. Only the OpenAI-compatible provider parse changes. Happy to send a PR.