fix(consolidation): eliminate duplicate observations (interleave dedup recall + tool & benchmark)

[nicoloboschi]

Problem

Consolidation could emit duplicate observations — multiple near-identical (sometimes byte-identical) observations of the same fact within a single document/session.

Root cause (investigated end-to-end)

Two distinct mechanisms, confirmed with the new per-bank LLM tracing (#1922):

1. Byte-identical creates — the LLM occasionally emits a CREATE whose text exactly matches an observation already shown to it (or an UPDATE in the same response).
2. Recall misses (the dominant residual) — the existing near-identical observation (the merge "twin") is semantic rank Fix dependencies and docker #1 for the new fact, but it shares no source-fact graph link and little lexical overlap, so RRF averaged it below the per-fact recall budget (consolidation_max_tokens=512, ~8 observations) and the LLM never saw it → created a duplicate. The cross-encoder demoted it the same way (semantic Fix dependencies and docker #1 → reranked async operations error #37).

Verified on traced runs: for every residual near-dup, the twin was semantic #1 (cosine 0.84–0.88) yet absent from the consolidation prompt; the LLM's own reason said "not covered by existing observations".

Fix

• Interleave fusion for consolidation dedup recall. A new round-robin fusion takes each retrieval arm's Fix dependencies and docker #1, then each rename to hindsight #2, … (semantic → bm25 → graph → temporal), de-duplicating, until the budget fills — so every arm's top hit is guaranteed a slot and the semantic-Fix dependencies and docker #1 twin is always shown. The LLM then UPDATEs instead of creating a duplicate.
• Unified reranking strategy param. Replaced the recall rerank: bool + ad-hoc passthrough with a single named strategy: cross_encoder (default, unchanged for user-facing recall), rrf, interleave. Consolidation uses interleave. For interleave the fusion order is authoritative — the recency/temporal re-sort is skipped (that re-sort is part of what buried the twin).
• Deterministic post-LLM reconciliation guard for the byte-identical case: a CREATE whose normalised text matches an observation shown to the LLM (or an in-response UPDATE) is dropped (no info loss).
• reason field on consolidation actions (LLM explains each create/update/delete) — kept for observability.

Results

Measured on an English translation of the hermes transcript (removes a cross-lingual embedding confound: the default bge-small-en can't align Chinese facts ↔ English observations, which inflated loose near-dups):

metric	RRF (before) 1/4	interleave (after) 1/4
near-dups @0.97	4 (4%)	0
near-dups @0.92	9 (9%)	0
coverage	89%	94%
both-flags / failures	0 / 0	0 / 0

No false merges: the heavy observations are coherent single themes, and distinct facets (per-language targets, plan phases) stay as separate observations.

Also included

• hindsight-dev observation-dedup tool (obs_dedup) — re-embeds observations, cosine-clusters near-dups; reused by the benchmark.
• benchmarks/obs quality benchmark — ingests a dataset, drains consolidation (serial SyncTaskBackend so the drain loop is the sole consolidator), and reports the duplication rate (exact + @0.97/@0.92). --fraction / --dataset / --wipe-bank flags.

Testing

• tests/test_interleave_fusion.py — round-robin order, dedup, semantic-top-hit-always-first, strictly-decreasing scores.
• tests/test_consolidation_dedup.py — the reconciliation guard.
• Default recall path unchanged (cross_encoder); existing recall/scoring tests pass.