Skip to content

Embedding model cold-loads on every prompt — add monolingual/bilingual language mode #108

Description

@aaronsb

Problem

Every UserPromptSubmit hook invocation spawns way-embed match as a subprocess — up to twice per prompt:

  1. EN corpus (21 MB GGUF, 132 entries)
  2. Multilingual corpus (126 MB GGUF, 1,411 entries)

Each invocation cold-loads the full GGUF model into memory, computes the query embedding, scores all corpus entries, then exits. The 126 MB multilingual model load causes a visible CPU spike (~0.4s, fans spin up) on every single user message.

The corpus itself is not rebuilding — corpus --if-stale correctly exits in ~1ms when the manifest is fresh. The cost is purely model loading at inference time.

Evidence

# Per-prompt cost (ways scan prompt):
0.35s user, 0.04s system — 103% cpu, 0.38s wall

# Model sizes:
minilm-l6-v2.gguf                      21 MB   (EN)
multilingual-minilm-l12-v2-q8.gguf    126 MB   (multilingual)

# Corpus sizes:
ways-corpus-en.jsonl                   132 entries
ways-corpus-multi.jsonl              1,411 entries

Call path

UserPromptSubmit hook
  → check-prompt.sh
    → ways scan prompt --query "..."
      → batch_embed_score()          [scoring.rs]
        → run_embed_match(en_corpus, en_model)       # spawn way-embed, load 21MB
        → run_embed_match(multi_corpus, multi_model)  # spawn way-embed, load 126MB

Proposal: Language mode config

Add a configuration property that controls which embedding models are loaded:

bilingual        → EN + multilingual models (current default, ~147 MB loaded per prompt)
monolingual:en   → EN model only (~21 MB per prompt)
monolingual:de   → multilingual model only, filtered to de corpus entries

Benefits

  • Monolingual EN users (majority case) skip the 126 MB multilingual model entirely — ~6x less memory, faster inference
  • Monolingual non-EN users skip the EN model, use only the multilingual model with their language's corpus subset
  • Bilingual preserves current behavior for users who actually need it
  • No cross-platform complexity (a daemon/socket approach would diverge across OS)

Where it lives

The setting could go in:

  • settings.json under a ways key (e.g., "ways": {"language_mode": "monolingual:en"})
  • Or as a frontmatter default in hooks/ways/core.md

Implementation sketch

  1. Add language_mode config reading to batch_embed_score() in scoring.rs
  2. Skip corpus/model pairs that don't match the configured mode
  3. Default to monolingual:en (matches the majority use case; bilingual users opt in)
  4. ways status should report the active language mode

Future consideration

A longer-term fix would keep way-embed resident (daemon with socket/pipe interface) so the model loads once per session rather than per prompt. However, this introduces cross-platform complexity (Unix sockets vs named pipes vs TCP) and process lifecycle management. The language mode approach is simpler and addresses the immediate pain.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions