Commit 0720da2
authored
fix(recall): over-fetch vector candidates and exclude artifact types (#205)
## Problem
A Claude Desktop session opened with a memory recall that was "basically
useless" — it returned 22 memories about Lennard/Berlin but **none**
were the context the user needed ("Lennard's situation we resolved"), so
the assistant concluded it had no shared history.
Reproduced against the **production corpus** (not a local config
glitch):
- The relevant memory exists and is high value: `ee830782` "Lennard
BAföG Mahnung Apr 2026" (919 chars, importance 0.8, tags
`bafoeg/mahnung/freistellung/stundung/legal/german-bureaucracy`).
- Query `"...Jobcenter social welfare resolved problem"` → it surfaces
at **rank #4**.
- Query `"...Berlin flatmate Bürgergeld Wohngeld"` → it is **absent**
from all 22 results, even though it's inside the time window and the
result count (22) was under the limit (25). It never entered the
candidate pool.
- The pool filled with noise: #1 was a floor-plan memory matched on
"Berlin flat" ≈ "Berlin flatmate"; ranks 10–22 were all
`match_type:keyword` on the bare token "berlin"; and three
importance-**0.000** `MetaPattern` artifacts leaked in.
## Root cause
1. **The recall miss.** The vector candidate pool was fetched at **1×
the requested limit**, with raw vector similarity as the sole gate
(`vector_fetch_limit = per_query_limit`). The richer final score
(importance, exact-match, tags) only re-ranks memories that *already
survived* that cut — so a high-importance, exact-topic memory that's a
slightly weaker pure-vector match is discarded before its signal can
lift it. A high-frequency token like "Berlin" saturates the pool and
evicts the relevant memory.
2. **The misleading padding.** `MetaPattern` consolidation artifacts
(importance 0.0, "cluster over 0 days") had no recall exclusion and
leaked in via bare-token keyword matches.
0.16.0 didn't create these mechanisms; its ranking release amplified
them.
## Fix
- **Over-fetch + re-rank** (`automem/api/recall.py`):
`vector_fetch_limit = min(per_query_limit * RECALL_VECTOR_OVERFETCH,
RECALL_VECTOR_FETCH_CAP)`. The existing re-rank
(`compute_metadata_score` → sort) then surfaces the right memory — **no
scoring-weight changes**. Results are still trimmed to the requested
limit (`_guarantee_priority_results`), so response size is unchanged.
`RECALL_VECTOR_OVERFETCH=1` restores legacy behavior.
- **Artifact exclusion** (`automem/search/runtime_recall_helpers.py`):
exclude `RECALL_EXCLUDED_TYPES` (default `{"MetaPattern"}`) at the
universal `_result_passes_filters` chokepoint (covers vector, keyword,
metadata, expansion, and state-filter paths) plus the graph/metadata
keyword Cypher so artifacts don't consume candidate slots.
New config (`automem/config.py`): `RECALL_VECTOR_OVERFETCH=4`,
`RECALL_VECTOR_FETCH_CAP=200`, `RECALL_EXCLUDED_TYPES=MetaPattern` — all
env-overridable.
## Tests (RED → GREEN)
- `tests/test_recall_overfetch.py` — drives `/recall` with a fake
Qdrant: asserts the widened fetch limit (20 vs 5), that the target
ranked past 1× is recalled and re-ranked into the top results, that the
response stays ≤ limit, and an off-switch (`OVERFETCH=1`) that
reproduces the miss.
- `tests/test_recall_artifact_exclusion.py` — `_result_passes_filters`
excludes `MetaPattern`/keeps normal types; `_graph_keyword_search` emits
the Cypher exclusion predicate + param.
- Full unit suite green: **662 passed, 12 skipped**. `black` + `flake8`
clean.
## Pre-merge validation (infra/deploy-gated — not run in this PR)
Candidate-pool sizing changes ranking, so gate the merge on the existing
harnesses (benchmark ownership stays in `automem`):
- [ ] **Recall Quality Lab**: `make lab-clone` → `make lab-queries` →
`make lab-test CONFIG=baseline` → `make lab-compare CONFIG=fix_v1
BASELINE=baseline`.
- [ ] **Sweep the multiplier** to confirm the default: `make lab-sweep
PARAM=RECALL_VECTOR_OVERFETCH VALUES=1,2,4,8` (latency vs recall@k —
Qdrant fetch cost scales with the multiplier).
- [ ] **LoCoMo** non-regressing: `make test-locomo` / `make
test-locomo-live`.
- [ ] **Live smoke** (post-deploy): re-run the transcript query and
confirm `ee830782` appears and `MetaPattern` rows are gone.
## Out of scope (follow-ups)
- **Entity unification**: "Lennard"/"Leonard" are split across spellings
and mis-typed (`entity:organizations:lennard`). Needs an
enrichment-validator fix + backfill migration.
- **Low-confidence signal**: recall returns weak results with no
"nothing strongly matched" cue (what misled the consumer). Consider
enabling/raising `RECALL_RELEVANCE_GATE` or surfacing the top evidence
score.
🤖 Generated with [Claude Code](https://claude.com/claude-code)9 files changed
Lines changed: 1035 additions & 93 deletions
File tree
- automem
- api
- search
- stores
- benchmarks
- docs
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
| 24 | + | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
25 | 28 | | |
26 | 29 | | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
| |||
1555 | 1559 | | |
1556 | 1560 | | |
1557 | 1561 | | |
1558 | | - | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
1559 | 1567 | | |
1560 | 1568 | | |
1561 | | - | |
| 1569 | + | |
1562 | 1570 | | |
1563 | 1571 | | |
1564 | 1572 | | |
| |||
1576 | 1584 | | |
1577 | 1585 | | |
1578 | 1586 | | |
| 1587 | + | |
| 1588 | + | |
1579 | 1589 | | |
1580 | 1590 | | |
1581 | 1591 | | |
| |||
1945 | 1955 | | |
1946 | 1956 | | |
1947 | 1957 | | |
| 1958 | + | |
1948 | 1959 | | |
1949 | 1960 | | |
1950 | | - | |
| 1961 | + | |
| 1962 | + | |
| 1963 | + | |
| 1964 | + | |
| 1965 | + | |
| 1966 | + | |
| 1967 | + | |
| 1968 | + | |
| 1969 | + | |
| 1970 | + | |
1951 | 1971 | | |
1952 | | - | |
| 1972 | + | |
| 1973 | + | |
| 1974 | + | |
| 1975 | + | |
1953 | 1976 | | |
1954 | 1977 | | |
1955 | 1978 | | |
1956 | 1979 | | |
1957 | 1980 | | |
1958 | 1981 | | |
1959 | | - | |
| 1982 | + | |
1960 | 1983 | | |
1961 | 1984 | | |
1962 | 1985 | | |
| |||
1973 | 1996 | | |
1974 | 1997 | | |
1975 | 1998 | | |
| 1999 | + | |
| 2000 | + | |
1976 | 2001 | | |
1977 | 2002 | | |
1978 | 2003 | | |
1979 | 2004 | | |
1980 | | - | |
| 2005 | + | |
1981 | 2006 | | |
1982 | 2007 | | |
1983 | 2008 | | |
| |||
1993 | 2018 | | |
1994 | 2019 | | |
1995 | 2020 | | |
| 2021 | + | |
| 2022 | + | |
| 2023 | + | |
| 2024 | + | |
| 2025 | + | |
1996 | 2026 | | |
1997 | 2027 | | |
1998 | 2028 | | |
1999 | 2029 | | |
2000 | | - | |
| 2030 | + | |
2001 | 2031 | | |
2002 | 2032 | | |
2003 | 2033 | | |
2004 | 2034 | | |
2005 | 2035 | | |
2006 | 2036 | | |
| 2037 | + | |
2007 | 2038 | | |
2008 | 2039 | | |
2009 | 2040 | | |
| 2041 | + | |
| 2042 | + | |
| 2043 | + | |
| 2044 | + | |
| 2045 | + | |
| 2046 | + | |
2010 | 2047 | | |
2011 | 2048 | | |
2012 | 2049 | | |
| |||
2023 | 2060 | | |
2024 | 2061 | | |
2025 | 2062 | | |
| 2063 | + | |
| 2064 | + | |
| 2065 | + | |
| 2066 | + | |
| 2067 | + | |
| 2068 | + | |
| 2069 | + | |
| 2070 | + | |
| 2071 | + | |
| 2072 | + | |
| 2073 | + | |
| 2074 | + | |
| 2075 | + | |
| 2076 | + | |
| 2077 | + | |
| 2078 | + | |
| 2079 | + | |
| 2080 | + | |
| 2081 | + | |
| 2082 | + | |
| 2083 | + | |
| 2084 | + | |
| 2085 | + | |
| 2086 | + | |
| 2087 | + | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
| 2095 | + | |
| 2096 | + | |
| 2097 | + | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
| 2102 | + | |
| 2103 | + | |
| 2104 | + | |
2026 | 2105 | | |
2027 | 2106 | | |
2028 | | - | |
2029 | | - | |
| 2107 | + | |
| 2108 | + | |
| 2109 | + | |
| 2110 | + | |
| 2111 | + | |
| 2112 | + | |
| 2113 | + | |
| 2114 | + | |
| 2115 | + | |
2030 | 2116 | | |
| 2117 | + | |
| 2118 | + | |
| 2119 | + | |
| 2120 | + | |
| 2121 | + | |
2031 | 2122 | | |
2032 | 2123 | | |
2033 | 2124 | | |
2034 | 2125 | | |
2035 | 2126 | | |
2036 | 2127 | | |
2037 | 2128 | | |
2038 | | - | |
| 2129 | + | |
2039 | 2130 | | |
2040 | 2131 | | |
2041 | 2132 | | |
| |||
2045 | 2136 | | |
2046 | 2137 | | |
2047 | 2138 | | |
| 2139 | + | |
| 2140 | + | |
2048 | 2141 | | |
2049 | | - | |
2050 | | - | |
2051 | | - | |
2052 | | - | |
2053 | | - | |
2054 | | - | |
2055 | | - | |
2056 | | - | |
2057 | | - | |
2058 | | - | |
2059 | | - | |
2060 | | - | |
2061 | | - | |
2062 | | - | |
2063 | | - | |
2064 | | - | |
2065 | | - | |
2066 | | - | |
2067 | | - | |
2068 | | - | |
2069 | | - | |
2070 | | - | |
2071 | | - | |
2072 | | - | |
2073 | | - | |
2074 | | - | |
2075 | | - | |
2076 | | - | |
2077 | | - | |
2078 | | - | |
2079 | | - | |
2080 | | - | |
2081 | | - | |
2082 | | - | |
2083 | 2142 | | |
2084 | 2143 | | |
2085 | 2144 | | |
| |||
2221 | 2280 | | |
2222 | 2281 | | |
2223 | 2282 | | |
2224 | | - | |
2225 | | - | |
2226 | | - | |
2227 | | - | |
2228 | | - | |
2229 | | - | |
2230 | | - | |
2231 | | - | |
2232 | | - | |
2233 | | - | |
2234 | | - | |
2235 | | - | |
2236 | | - | |
2237 | | - | |
| 2283 | + | |
| 2284 | + | |
| 2285 | + | |
| 2286 | + | |
| 2287 | + | |
| 2288 | + | |
| 2289 | + | |
| 2290 | + | |
| 2291 | + | |
| 2292 | + | |
| 2293 | + | |
| 2294 | + | |
| 2295 | + | |
2238 | 2296 | | |
2239 | 2297 | | |
2240 | 2298 | | |
| |||
2372 | 2430 | | |
2373 | 2431 | | |
2374 | 2432 | | |
| 2433 | + | |
| 2434 | + | |
| 2435 | + | |
| 2436 | + | |
2375 | 2437 | | |
2376 | 2438 | | |
2377 | 2439 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
150 | 167 | | |
151 | 168 | | |
152 | 169 | | |
| |||
0 commit comments