Skip to content

Commit 0720da2

Browse files
authored
fix(recall): over-fetch vector candidates and exclude artifact types (#205)
## Problem A Claude Desktop session opened with a memory recall that was "basically useless" — it returned 22 memories about Lennard/Berlin but **none** were the context the user needed ("Lennard's situation we resolved"), so the assistant concluded it had no shared history. Reproduced against the **production corpus** (not a local config glitch): - The relevant memory exists and is high value: `ee830782` "Lennard BAföG Mahnung Apr 2026" (919 chars, importance 0.8, tags `bafoeg/mahnung/freistellung/stundung/legal/german-bureaucracy`). - Query `"...Jobcenter social welfare resolved problem"` → it surfaces at **rank #4**. - Query `"...Berlin flatmate Bürgergeld Wohngeld"` → it is **absent** from all 22 results, even though it's inside the time window and the result count (22) was under the limit (25). It never entered the candidate pool. - The pool filled with noise: #1 was a floor-plan memory matched on "Berlin flat" ≈ "Berlin flatmate"; ranks 10–22 were all `match_type:keyword` on the bare token "berlin"; and three importance-**0.000** `MetaPattern` artifacts leaked in. ## Root cause 1. **The recall miss.** The vector candidate pool was fetched at **1× the requested limit**, with raw vector similarity as the sole gate (`vector_fetch_limit = per_query_limit`). The richer final score (importance, exact-match, tags) only re-ranks memories that *already survived* that cut — so a high-importance, exact-topic memory that's a slightly weaker pure-vector match is discarded before its signal can lift it. A high-frequency token like "Berlin" saturates the pool and evicts the relevant memory. 2. **The misleading padding.** `MetaPattern` consolidation artifacts (importance 0.0, "cluster over 0 days") had no recall exclusion and leaked in via bare-token keyword matches. 0.16.0 didn't create these mechanisms; its ranking release amplified them. ## Fix - **Over-fetch + re-rank** (`automem/api/recall.py`): `vector_fetch_limit = min(per_query_limit * RECALL_VECTOR_OVERFETCH, RECALL_VECTOR_FETCH_CAP)`. The existing re-rank (`compute_metadata_score` → sort) then surfaces the right memory — **no scoring-weight changes**. Results are still trimmed to the requested limit (`_guarantee_priority_results`), so response size is unchanged. `RECALL_VECTOR_OVERFETCH=1` restores legacy behavior. - **Artifact exclusion** (`automem/search/runtime_recall_helpers.py`): exclude `RECALL_EXCLUDED_TYPES` (default `{"MetaPattern"}`) at the universal `_result_passes_filters` chokepoint (covers vector, keyword, metadata, expansion, and state-filter paths) plus the graph/metadata keyword Cypher so artifacts don't consume candidate slots. New config (`automem/config.py`): `RECALL_VECTOR_OVERFETCH=4`, `RECALL_VECTOR_FETCH_CAP=200`, `RECALL_EXCLUDED_TYPES=MetaPattern` — all env-overridable. ## Tests (RED → GREEN) - `tests/test_recall_overfetch.py` — drives `/recall` with a fake Qdrant: asserts the widened fetch limit (20 vs 5), that the target ranked past 1× is recalled and re-ranked into the top results, that the response stays ≤ limit, and an off-switch (`OVERFETCH=1`) that reproduces the miss. - `tests/test_recall_artifact_exclusion.py` — `_result_passes_filters` excludes `MetaPattern`/keeps normal types; `_graph_keyword_search` emits the Cypher exclusion predicate + param. - Full unit suite green: **662 passed, 12 skipped**. `black` + `flake8` clean. ## Pre-merge validation (infra/deploy-gated — not run in this PR) Candidate-pool sizing changes ranking, so gate the merge on the existing harnesses (benchmark ownership stays in `automem`): - [ ] **Recall Quality Lab**: `make lab-clone` → `make lab-queries` → `make lab-test CONFIG=baseline` → `make lab-compare CONFIG=fix_v1 BASELINE=baseline`. - [ ] **Sweep the multiplier** to confirm the default: `make lab-sweep PARAM=RECALL_VECTOR_OVERFETCH VALUES=1,2,4,8` (latency vs recall@k — Qdrant fetch cost scales with the multiplier). - [ ] **LoCoMo** non-regressing: `make test-locomo` / `make test-locomo-live`. - [ ] **Live smoke** (post-deploy): re-run the transcript query and confirm `ee830782` appears and `MetaPattern` rows are gone. ## Out of scope (follow-ups) - **Entity unification**: "Lennard"/"Leonard" are split across spellings and mis-typed (`entity:organizations:lennard`). Needs an enrichment-validator fix + backfill migration. - **Low-confidence signal**: recall returns weak results with no "nothing strongly matched" cue (what misled the consumer). Consider enabling/raising `RECALL_RELEVANCE_GATE` or surfacing the top evidence score. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents f207a40 + a96478a commit 0720da2

9 files changed

Lines changed: 1035 additions & 93 deletions

File tree

automem/api/recall.py

Lines changed: 120 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -14,16 +14,20 @@
1414
DEFAULT_EXPAND_RELATIONS,
1515
FILTERABLE_RELATIONS,
1616
RECALL_ADAPTIVE_FLOOR,
17+
RECALL_EXCLUDED_TYPES,
1718
RECALL_EXPANSION_LIMIT,
1819
RECALL_METADATA_SEARCH_ENABLED,
1920
RECALL_MIN_SCORE,
2021
RECALL_RECENCY_BIAS,
2122
RECALL_RELATION_LIMIT,
23+
RECALL_VECTOR_FETCH_CAP,
24+
RECALL_VECTOR_OVERFETCH,
2225
SEARCH_WEIGHT_TEMPORAL,
2326
canonicalize_relation_type,
2427
expand_relation_query_types,
2528
normalize_relation_type,
2629
)
30+
from automem.search.runtime_recall_helpers import _hydrate_vector_relations
2731
from automem.utils.graph import _serialize_node
2832
from automem.utils.time import _parse_iso_datetime, query_has_temporal_intent
2933

@@ -1555,10 +1559,14 @@ def _expand_related_memories(
15551559
seed_score = float(seed.get("final_score", seed.get("score", 0.0)) or 0.0)
15561560

15571561
try:
1558-
rel_filter = " AND type(r) IN $types" if query_relation_types else ""
1562+
where_clauses = ["m.id <> related.id"]
1563+
if query_relation_types:
1564+
where_clauses.append("type(r) IN $types")
1565+
if RECALL_EXCLUDED_TYPES:
1566+
where_clauses.append("NOT coalesce(related.type, '') IN $excluded_types")
15591567
query = f"""
15601568
MATCH (m:Memory {{id: $id}})-[r]-(related:Memory)
1561-
WHERE m.id <> related.id{rel_filter}
1569+
WHERE {' AND '.join(where_clauses)}
15621570
RETURN type(r) as relation_type,
15631571
coalesce(
15641572
r.strength,
@@ -1576,6 +1584,8 @@ def _expand_related_memories(
15761584
params: Dict[str, Any] = {"id": seed_id, "limit": per_seed_limit}
15771585
if query_relation_types:
15781586
params["types"] = query_relation_types
1587+
if RECALL_EXCLUDED_TYPES:
1588+
params["excluded_types"] = list(RECALL_EXCLUDED_TYPES)
15791589
records = graph.query(query, params)
15801590
except Exception:
15811591
logger.exception("Failed to expand relations for seed %s", seed_id)
@@ -1945,18 +1955,31 @@ def _run_single_query(
19451955

19461956
local_results: List[Dict[str, Any]] = []
19471957
vector_matches: List[Dict[str, Any]] = []
1958+
vector_seen: set[str] = set(local_seen)
19481959

19491960
if qdrant_client is not None:
1950-
vector_fetch_limit = per_query_limit
1961+
# Over-fetch vector candidates so the richer final scoring
1962+
# (importance, exact-match, tags) re-ranks a wide enough pool. A
1963+
# high-importance, exact-topic memory that is a slightly weaker
1964+
# pure-vector match would otherwise be cut before its signal can
1965+
# lift it. Results are trimmed to `limit` downstream, so the
1966+
# response size is unchanged. OVERFETCH=1 restores legacy behavior.
1967+
vector_fetch_limit = max(
1968+
per_query_limit,
1969+
min(per_query_limit * RECALL_VECTOR_OVERFETCH, RECALL_VECTOR_FETCH_CAP),
1970+
)
19511971
if tag_filters and (query_str or embedding_param):
1952-
vector_fetch_limit = max(per_query_limit, recall_max_limit)
1972+
vector_fetch_limit = max(
1973+
per_query_limit,
1974+
min(max(vector_fetch_limit, recall_max_limit), RECALL_VECTOR_FETCH_CAP),
1975+
)
19531976
vector_matches = vector_search(
19541977
qdrant_client,
19551978
graph,
19561979
query_str,
19571980
embedding_param,
19581981
vector_fetch_limit,
1959-
local_seen,
1982+
vector_seen,
19601983
tag_filters,
19611984
tag_mode,
19621985
tag_match,
@@ -1973,11 +1996,13 @@ def _run_single_query(
19731996

19741997
remaining_slots = max(0, per_query_limit - len(local_results))
19751998
if remaining_slots and graph is not None:
1999+
graph_seen = set(local_seen)
2000+
graph_seen.update(vector_seen)
19762001
graph_matches = graph_keyword_search(
19772002
graph,
19782003
query_str,
19792004
remaining_slots,
1980-
local_seen,
2005+
graph_seen,
19812006
start_time=start_time,
19822007
end_time=end_time,
19832008
tag_filters=tag_filters,
@@ -1993,20 +2018,32 @@ def _run_single_query(
19932018
and RECALL_METADATA_SEARCH_ENABLED
19942019
):
19952020
metadata_slots = max(1, min(per_query_limit, 10))
2021+
metadata_upgrade_ids = {
2022+
memory_id
2023+
for memory_id in (_result_memory_id(result) for result in local_results)
2024+
if memory_id
2025+
}
19962026
metadata_matches = metadata_keyword_search(
19972027
graph,
19982028
query_str,
19992029
metadata_slots,
2000-
local_seen,
2030+
set(metadata_upgrade_ids),
20012031
start_time=start_time,
20022032
end_time=end_time,
20032033
tag_filters=tag_filters,
20042034
tag_mode=tag_mode,
20052035
tag_match=tag_match,
20062036
exclude_tags=exclude_tags,
2037+
include_seen_ids=metadata_upgrade_ids,
20072038
)
20082039
local_results.extend(metadata_matches)
20092040

2041+
local_seen.update(
2042+
str(result.get("id") or (result.get("memory") or {}).get("id") or "")
2043+
for result in local_results
2044+
if result.get("id") or (result.get("memory") or {}).get("id")
2045+
)
2046+
20102047
tags_only_request = (
20112048
not query_str
20122049
and not (embedding_param and embedding_param.strip())
@@ -2023,19 +2060,73 @@ def _run_single_query(
20232060
)
20242061
local_results.extend(tag_only_results)
20252062

2063+
query_tokens = extract_keywords(query_str.lower()) if query_str else []
2064+
2065+
def _rank_local_results(results: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
2066+
for result in results:
2067+
final_score, components = compute_metadata_score(
2068+
result,
2069+
query_str or "",
2070+
query_tokens,
2071+
context_profile,
2072+
)
2073+
score_components = result.setdefault("score_components", {})
2074+
score_components.update(components)
2075+
result["final_score"] = final_score
2076+
if "original_score" not in result:
2077+
result["original_score"] = result.get("score", 0.0)
2078+
result["score"] = final_score
2079+
2080+
ranked = [
2081+
res
2082+
for res in results
2083+
if result_passes_filters(
2084+
res, start_time, end_time, tag_filters, tag_mode, tag_match, exclude_tags
2085+
)
2086+
]
2087+
2088+
if min_score is not None and min_score > 0:
2089+
ranked = [res for res in ranked if float(res.get("final_score", 0.0)) >= min_score]
2090+
2091+
ranked, _local_dedup_removed = _dedupe_results(ranked)
2092+
2093+
if sort_param == "score":
2094+
ranked.sort(key=_score_sort_key)
2095+
elif sort_param in {"time_desc", "time_asc"}:
2096+
ranked.sort(key=_time_sort_key, reverse=(sort_param == "time_desc"))
2097+
elif sort_param in {"updated_desc", "updated_asc"}:
2098+
# Same key, but favor updated_at (already primary) and keep the name explicit for callers
2099+
ranked.sort(key=_time_sort_key, reverse=(sort_param == "updated_desc"))
2100+
2101+
return ranked
2102+
2103+
local_results = _rank_local_results(local_results)
2104+
20262105
context_injected = False
20272106
if context_profile:
2028-
priority_ids_missing = not _results_have_priority_ids(local_results, context_profile)
2029-
needs_context_injection = not _results_have_priority(local_results, context_profile)
2107+
returnable_results = _guarantee_priority_results(
2108+
local_results, context_profile, per_query_limit
2109+
)
2110+
priority_ids_missing = not _results_have_priority_ids(
2111+
returnable_results, context_profile
2112+
)
2113+
needs_context_injection = not _results_have_priority(
2114+
returnable_results, context_profile
2115+
)
20302116
if priority_ids_missing or needs_context_injection:
2117+
injection_seen = {
2118+
memory_id
2119+
for memory_id in (_result_memory_id(result) for result in returnable_results)
2120+
if memory_id
2121+
}
20312122
context_injected = _inject_priority_memories(
20322123
local_results,
20332124
graph,
20342125
qdrant_client,
20352126
graph_keyword_search,
20362127
vector_filter_only_tag_search,
20372128
context_profile,
2038-
local_seen,
2129+
injection_seen,
20392130
result_passes_filters,
20402131
start_time,
20412132
end_time,
@@ -2045,41 +2136,9 @@ def _run_single_query(
20452136
logger,
20462137
exclude_tags,
20472138
)
2139+
if context_injected:
2140+
local_results = _rank_local_results(local_results)
20482141

2049-
query_tokens = extract_keywords(query_str.lower()) if query_str else []
2050-
for result in local_results:
2051-
final_score, components = compute_metadata_score(
2052-
result,
2053-
query_str or "",
2054-
query_tokens,
2055-
context_profile,
2056-
)
2057-
result.setdefault("score_components", components)
2058-
result["score_components"].update(components)
2059-
result["final_score"] = final_score
2060-
result["original_score"] = result.get("score", 0.0)
2061-
result["score"] = final_score
2062-
2063-
local_results = [
2064-
res
2065-
for res in local_results
2066-
if result_passes_filters(
2067-
res, start_time, end_time, tag_filters, tag_mode, tag_match, exclude_tags
2068-
)
2069-
]
2070-
2071-
if min_score is not None and min_score > 0:
2072-
local_results = [
2073-
res for res in local_results if float(res.get("final_score", 0.0)) >= min_score
2074-
]
2075-
2076-
if sort_param == "score":
2077-
local_results.sort(key=_score_sort_key)
2078-
elif sort_param in {"time_desc", "time_asc"}:
2079-
local_results.sort(key=_time_sort_key, reverse=(sort_param == "time_desc"))
2080-
elif sort_param in {"updated_desc", "updated_asc"}:
2081-
# Same key, but favor updated_at (already primary) and keep the name explicit for callers
2082-
local_results.sort(key=_time_sort_key, reverse=(sort_param == "updated_desc"))
20832142
local_results = _guarantee_priority_results(local_results, context_profile, per_query_limit)
20842143

20852144
# Track originating query for debugging/clients
@@ -2221,20 +2280,19 @@ def _run_single_query(
22212280
entity_tag_filters = tag_filters if expand_respect_tags else None
22222281
entity_tag_mode = tag_mode if expand_respect_tags else "any"
22232282
entity_tag_match = tag_match if expand_respect_tags else "exact"
2224-
if start_time or end_time or entity_tag_filters or exclude_tags:
2225-
entity_expansion_results = [
2226-
r
2227-
for r in entity_expansion_results
2228-
if result_passes_filters(
2229-
r,
2230-
start_time,
2231-
end_time,
2232-
entity_tag_filters,
2233-
entity_tag_mode,
2234-
entity_tag_match,
2235-
exclude_tags,
2236-
)
2237-
]
2283+
entity_expansion_results = [
2284+
r
2285+
for r in entity_expansion_results
2286+
if result_passes_filters(
2287+
r,
2288+
start_time,
2289+
end_time,
2290+
entity_tag_filters,
2291+
entity_tag_mode,
2292+
entity_tag_match,
2293+
exclude_tags,
2294+
)
2295+
]
22382296
results = seed_results + expansion_results + entity_expansion_results
22392297

22402298
state_filter_info: Optional[Dict[str, Any]] = None
@@ -2372,6 +2430,10 @@ def _run_single_query(
23722430
logger=logger,
23732431
)
23742432

2433+
# Hydrate vector relations after trimming/selection so over-fetched candidates
2434+
# do not fan out into graph relation queries before most of them are dropped.
2435+
_hydrate_vector_relations(graph, results, logger)
2436+
23752437
# Hydrate after scope-fallback fills so appended results get summaries too
23762438
_hydrate_missing_summaries_from_graph(results, graph, logger)
23772439

automem/config.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,23 @@
147147
"no",
148148
}
149149

150+
# Recall candidate-pool over-fetch. The vector store returns its top-K by raw
151+
# similarity, but the richer final score (importance, exact-match, tags) runs
152+
# only on those survivors — so a high-importance, exact-topic memory that is a
153+
# slightly weaker pure-vector match gets cut before its signal can lift it.
154+
# Over-fetch widens the pool the re-rank sees; the response is still trimmed to
155+
# the requested limit. Set OVERFETCH=1 to restore the legacy 1x behavior.
156+
RECALL_VECTOR_OVERFETCH = max(1, int(os.getenv("RECALL_VECTOR_OVERFETCH", "4")))
157+
# Absolute ceiling on the over-fetched candidate count, kept separate from the
158+
# user-facing RECALL_MAX_LIMIT so over-fetch isn't strangled by the response cap.
159+
RECALL_VECTOR_FETCH_CAP = max(1, int(os.getenv("RECALL_VECTOR_FETCH_CAP", "200")))
160+
161+
# Internal artifact memory types that must never surface in user-facing /recall
162+
# results (e.g. consolidation cluster summaries). Comma-separated env override.
163+
RECALL_EXCLUDED_TYPES = frozenset(
164+
t.strip() for t in os.getenv("RECALL_EXCLUDED_TYPES", "MetaPattern").split(",") if t.strip()
165+
)
166+
150167
# Memory content size limits (governs auto-summarization on store)
151168
# Soft limit: Content above this triggers auto-summarization
152169
MEMORY_CONTENT_SOFT_LIMIT = int(os.getenv("MEMORY_CONTENT_SOFT_LIMIT", "500"))

0 commit comments

Comments
 (0)