Summary
Add support for Ollama and other local embedding models so users can run Distill without sending data to external APIs.
Motivation
- Privacy-sensitive users and enterprises can't send data to OpenAI
- Local models eliminate API costs entirely
- Ollama is the most popular local model runner (100K+ GitHub stars)
- Positions Distill as a fully self-contained, air-gapped solution
Embedding providers to add
| Provider |
Interface |
Priority |
| Ollama |
HTTP (/api/embeddings) |
P0 |
| HuggingFace TEI |
HTTP (OpenAI-compatible) |
P1 |
| Sentence Transformers |
Python subprocess |
P2 |
Implementation
// pkg/embedding/ollama/client.go
type Config struct {
Host string // default: http://localhost:11434
Model string // default: nomic-embed-text
}
type Client struct { ... }
func (c *Client) Embed(ctx context.Context, text string) ([]float32, error)
func (c *Client) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
CLI usage
# Use Ollama for embeddings
distill api --embedding-provider ollama --embedding-model nomic-embed-text
# Use with config file
# distill.yaml
embedding:
provider: "ollama"
model: "nomic-embed-text"
host: "http://localhost:11434"
Config file support
Update pkg/config/config.go to support provider field in embedding config.
Deliverables
Acceptance Criteria
Summary
Add support for Ollama and other local embedding models so users can run Distill without sending data to external APIs.
Motivation
Embedding providers to add
/api/embeddings)Implementation
CLI usage
Config file support
Update
pkg/config/config.goto supportproviderfield in embedding config.Deliverables
pkg/embedding/ollama/client.go- Ollama HTTP clientpkg/embedding/ollama/client_test.go- Unit tests (mock HTTP)pkg/embedding/provider.go- Provider interface + factory functioncmd/api.goandcmd/serve.goto accept--embedding-providerflagpkg/config/config.gowith provider fielddocker-compose.ymlexample with Ollama sidecarAcceptance Criteria
distill api --embedding-provider ollamaworks with local Ollama