feat(consolidation): semantic dedup of near-duplicate observations (create + update)

[nicoloboschi]

Problem

Weak consolidation models (e.g. gemini-2.5-flash-lite) emit near-duplicate observations even when the twin is shown to them. Interleave-fusion recall (#1907) fixed this for strong models, but at 1/4 scale on the English hermes benchmark flash-lite still left ~7% residual near-duplicates (cosine ≥0.97).

Root cause

Instrumented the consolidation create path (HNSW probe vs an exact Python cosine over every observation) and ran flash-lite at 1/4 scale: 0 probe misses across the entire run — the create-time probe never fails to surface an existing twin. The residual duplicates are instead born on the UPDATE path: _execute_update_action rewrites an observation's text and re-embeds it to fold in a new fact, and the rewritten vector can drift to ≥0.97 of a different existing observation. Nothing reconciled that — dedup only ran on CREATE. Every residual pair had ≥1 member that had been updated (history>0/proof_count>1).

It is not a recall, HNSW-recall, pgvector-freshness, transaction-visibility, or caching problem (all ruled out with data: deterministic embeddings, autocommit connections, per-bank HNSW index present from row 0).

Fix

consolidation_dedup_threshold (default 1.0 = disabled, Postgres only). When set below 1.0, an observation that is ≥ the threshold cosine to an existing one is reconciled via a focused 1-by-1 LLM "merge or keep" call (anchored on the observation text — the correct obs↔obs comparison — scope consolidation_dedup; the model reads both texts so a number/negation/entity difference is respected):

• CREATE path — on "merge", fold the new source facts + synthesized text into the existing twin and skip the insert.
• UPDATE path (the fix for the residual) — after the rewrite+re-embed, probe the new vector (excluding the row itself); on "merge", fold the updated observation's sources into the twin, persist the merged text, and delete the now-redundant updated row.

Shared probe+adjudicate logic is factored into _dedup_adjudicate; _execute_update_action returns its freshly-computed embedding so the update-path probe doesn't re-embed.

Result

English hermes obs benchmark, flash-lite, 1/4 scale: residual ≥0.97 near-dups ~7% → 0–1% (UPDATE-path merges observed firing alongside CREATE-path merges). Default 1.0 keeps behaviour unchanged.

Tests

tests/test_consolidation_dedup.py (13, no LLM/DB): exact-text guard, create-path (no-twin / keep / merge-folds / picks-highest), update-path (merge folds+deletes the updated row / keep / self-exclusion / no-twin).