Switch embedding inference to ONNX Runtime and update all docs

- Replace PyTorch with ONNX Runtime for sentence-transformers inference
  (~50MB vs ~5GB, faster pod startup for KEDA scaling)
- Update dictionary with ONNX Runtime entry
- Update architecture, classification, demo guide with ONNX details
- Update implementation plan with Stage 13
- Update journal with ONNX switch details

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: johnmathews

CLAUDE.md

@@ -11,7 +11,7 @@ loan documents.
 - **PDF generation:** fpdf2 + Faker (nl_NL locale)
 - **Text extraction:** PyMuPDF (fitz)
 - **Rule-based classification:** Weighted keyword scoring (privacy level)
-- **Semantic classification:** sentence-transformers + anchor embeddings (environmental impact, industries)
+- **Semantic classification:** sentence-transformers + ONNX Runtime + anchor embeddings (environmental impact, industries)
 - **Testing:** pytest + coverage
 - **Linting:** ruff
 - **Deps:** uv

docs/architecture.md

@@ -154,10 +154,10 @@ storing → completed).
 | Aspect | Detail |
 |---|---|
 | Rule-based | Weighted keyword scoring → Privacy level (Public/Confidential/Secret) |
-| Semantic | sentence-transformers embedding → Environmental impact, Industry sectors, Privacy |
+| Semantic | sentence-transformers + ONNX Runtime embedding → Environmental impact, Industry sectors, Privacy |
 | Input | Redis stream `extracted` |
 | Output | Redis stream `classified` |
-| Model | all-MiniLM-L6-v2 (384 dimensions, ~100ms per document) |
+| Model | all-MiniLM-L6-v2 (384 dimensions, ~100ms per document, ONNX backend) |
 | Scaling | KEDA ScaledObject watching `extracted` stream depth |
 
 ### Store Workers (`src/worker/store.py`)
@@ -291,6 +291,8 @@ gitignored).
    language paragraphs describing each category. The embedding model captures
    meaning, enabling detection of concepts expressed in different words.
 
-6. **Local sentence-transformers over Azure OpenAI.** No API dependency, free,
-   runs anywhere. For production, Azure OpenAI text-embedding-3-small would
+6. **Local sentence-transformers with ONNX Runtime over Azure OpenAI.** No API
+   dependency, free, runs anywhere. ONNX Runtime replaces PyTorch for inference,
+   reducing container image size and memory usage (~50MB runtime vs ~5GB for
+   full PyTorch). For production, Azure OpenAI text-embedding-3-small would
    provide higher quality embeddings within Azure's data boundary.

docs/classification.md

@@ -184,8 +184,11 @@ We use pgvector (PostgreSQL extension) for the demo. For production at a bank,
 
 ### Embedding model
 
-We use `all-MiniLM-L6-v2` (sentence-transformers, 384 dimensions) — free,
-runs locally, no API dependency. For production:
+We use `all-MiniLM-L6-v2` (sentence-transformers, 384 dimensions) with **ONNX Runtime**
+as the inference backend instead of PyTorch. This reduces the runtime from ~5GB (full
+PyTorch) to ~50MB (ONNX Runtime), which means faster container image pulls, faster pod
+startup when KEDA scales workers, and lower memory per pod. Free, runs locally, no API
+dependency. For production:
 
 **Azure OpenAI `text-embedding-3-small`** would be preferred:
 - Higher quality embeddings (1536 dimensions)

docs/demo-guide.md

@@ -205,14 +205,17 @@ for RAG workloads on Azure.
 > production, if throughput requirements grew beyond what a single Redis instance
 > handles, I'd consider Azure Event Hubs, which is Kafka-compatible and fully managed."
 
-### Sentence-Transformers vs Azure OpenAI
-**When to mention:** When asked about the embedding model.
+### Sentence-Transformers + ONNX Runtime vs Azure OpenAI
+**When to mention:** When asked about the embedding model or container optimization.
 
 **What to say:**
-> "I used sentence-transformers locally — specifically all-MiniLM-L6-v2 — because
-> it runs without API dependencies and is free. For production, I'd use Azure OpenAI's
-> text-embedding-3-small model. The embeddings are higher quality, and the data stays
-> within Azure's boundary — important for a bank's compliance requirements."
+> "I used sentence-transformers locally — specifically all-MiniLM-L6-v2 — with ONNX
+> Runtime as the inference backend instead of PyTorch. ONNX Runtime is about 50MB
+> versus 5GB for full PyTorch, which makes a big difference in a K8s environment —
+> faster image pulls, faster pod startup when KEDA scales up, and lower memory per
+> worker. For production, I'd use Azure OpenAI's text-embedding-3-small model. The
+> embeddings are higher quality, and the data stays within Azure's boundary —
+> important for a bank's compliance requirements."
 
 ### Chaos Engineering
 **When to mention:** When they seem impressed by the self-healing demo.

docs/dictionary.md

@@ -401,6 +401,19 @@ concept — industrial pollution.
 We use the `all-MiniLM-L6-v2` model (sentence-transformers) which produces 384-dimensional
 vectors. "384-dimensional" means each text becomes a list of 384 numbers.
 
+### ONNX Runtime
+An optimized inference engine for running machine learning models. Instead of using
+full PyTorch (~5GB) to run the embedding model, we export the model to ONNX format
+and run it with ONNX Runtime (~50MB). Same model, same results, much smaller footprint.
+
+sentence-transformers supports this with a single parameter:
+```python
+SentenceTransformer("all-MiniLM-L6-v2", backend="onnx")
+```
+
+This is important for K8s because smaller images mean faster pod startup (KEDA can
+scale workers up more quickly) and lower memory usage per pod.
+
 ### Cosine Similarity
 A way to measure how similar two vectors are. Returns a value between -1 and 1:
 - 1.0 = identical meaning

docs/implementation-plan.md

@@ -25,6 +25,7 @@
 | 11 | Polish and demo rehearsal | MUST | 1.5-2h | TODO |
 | -- | **Day 4: Enhancements** | -- | -- | **DONE** |
 | 12 | Azure Blob Storage integration | HIGH | 2h | DONE (PDFs stored in Azure, metrics in Grafana) |
+| 13 | Switch to ONNX Runtime | MEDIUM | 15min | DONE (sentence-transformers backend="onnx", ~50MB vs ~5GB PyTorch) |
 
 **If time runs short:** Cut from the bottom. Stages 0-5 + 11 are non-negotiable. Stage 6 (Grafana) is
 the most important "nice to have" because it's the visual centerpiece of the demo. Stages 7-10 can

journal/260401-azure-blob-storage-and-metrics.md

@@ -84,6 +84,15 @@ to the Grafana dashboard, completing the storage layer of the pipeline.
 - `grafana/documentstream-dashboard.json` — 2 new blob storage panels
 - `tests/test_store.py` — 9 new tests (doc_type inference, blob upload, record fields)
 
+### Switch to ONNX Runtime
+
+- Changed `SentenceTransformer(MODEL_NAME)` to `SentenceTransformer(MODEL_NAME, backend="onnx")`
+  in `src/worker/semantic.py`
+- Replaced `sentence-transformers` dependency with `sentence-transformers[onnx]` in `pyproject.toml`
+  (pulls in `onnxruntime` and `optimum`)
+- ONNX Runtime is ~50MB vs ~5GB for full PyTorch — faster image pulls, faster KEDA scale-up,
+  lower memory per classify-worker pod
+
 ## Test Status
 
 - 92 tests passing (up from 83), lint clean

pyproject.toml

@@ -11,7 +11,7 @@ dependencies = [
     "pymupdf>=1.25.0",
     "redis>=5.2.0",
     "psycopg[binary]>=3.2.0",
-    "sentence-transformers>=4.1.0",
+    "sentence-transformers[onnx]>=4.1.0",
     "jinja2>=3.1.0",
     "python-multipart>=0.0.20",
     "azure-storage-blob>=12.24.0",

src/worker/semantic.py

@@ -38,7 +38,7 @@
 @lru_cache(maxsize=1)
 def _get_model() -> SentenceTransformer:
     """Load the embedding model (cached, loaded once)."""
-    return SentenceTransformer(MODEL_NAME)
+    return SentenceTransformer(MODEL_NAME, backend="onnx")
 
 
 def embed_text(text: str) -> NDArray[np.float32]: