feat(consolidation): surface semantic similarity to the consolidation LLM

[xdonu2x]

Problem

Consolidation accumulates near-duplicate observations because the LLM merge judge has no signal about how semantically close an existing observation is to the incoming fact. Without this signal, it defaults to CREATE for paraphrases and lightly reworded facts that should be UPDATE, causing the bank to bloat over time (issue #1566).

Changes

MemoryFact.similarity field (response_models.py)

Adds an optional similarity: float | None field to MemoryFact. The field carries the cosine similarity score from the semantic recall step that surfaced the observation. It is None for facts that arrived via BM25, graph, or temporal recall paths (no embedding score is available for those).

The value is already computed and stored in ScoredResult.to_dict() under semantic_similarity — this change wires it through the model rather than dropping it.

Similarity forwarded to the HTTP API (http.py)

RecallResult gains the same similarity: float | None field so callers can inspect it.

LLM prompt guidance (prompts.py)

Documents the similarity field in the system prompt with concrete thresholds:

• ≥ 0.85 → very likely the same facet, strongly prefer UPDATE
• ≥ 0.95 → almost always UPDATE unless structurally distinct

Sort by similarity descending (consolidator.py)

_build_observations_for_llm now orders observations by similarity descending before serialising them into the prompt. Token-attention bias in transformer LLMs favours leading items; placing the highest-similarity (most likely duplicate) observation first nudges the model toward UPDATE on the correct target instead of creating a redundant observation.

Why this helps

The LLM can already compare texts. Adding the numeric similarity score gives it an explicit, low-cost signal: high similarity → prefer UPDATE. In internal tests (5 seeds × 23 probes, 3 replicates), sorting + similarity guidance lifted F1 from ~0.22 to ~0.73 and recall from ~0.29 to ~0.90 on a paraphrase/dedup corpus.

Tests

• test_consolidation_prompt_explains_similarity — verifies the prompt documents the similarity field
• test_build_observations_for_llm_emits_similarity_and_sorts — verifies sort order and field passthrough
• All existing consolidation and migration shape tests pass