fix(recall): over-fetch vector candidates and exclude artifact types#205
Conversation
Recall could drop the single most relevant memory while padding results
with internal artifacts. Two compounding causes, both reproduced against
the production corpus (query "Lennard social benefits situation Berlin
flatmate ..." returned 22 results that omitted the one relevant memory
and leaked importance-0.0 MetaPattern rows):
1. The vector candidate pool was fetched at 1x the requested limit, with
raw vector similarity as the sole gate. The richer final score
(importance, exact-match, tags) only re-ranks survivors of that cut, so
a high-importance, exact-topic memory that is a slightly weaker pure-
vector match was discarded before its signal could lift it; a high-
frequency query token ("Berlin") saturates the pool and evicts it.
Fix: over-fetch the vector pool (RECALL_VECTOR_OVERFETCH, default 4,
capped by RECALL_VECTOR_FETCH_CAP=200) so the existing re-rank sees a
wide enough candidate set. Response is still trimmed to the limit.
2. MetaPattern consolidation artifacts had no recall exclusion and leaked
in via bare-token keyword matches. Fix: exclude RECALL_EXCLUDED_TYPES
(default {"MetaPattern"}) at the universal _result_passes_filters
chokepoint plus the graph/metadata keyword Cypher.
Adds RED->GREEN coverage in tests/test_recall_overfetch.py and
tests/test_recall_artifact_exclusion.py. Full unit suite green (662 passed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ddc9ecce32
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
## Summary - Preserve the caller's requested vector fetch limit even when `RECALL_VECTOR_FETCH_CAP` is configured below `limit`. - Add a regression test for the cap-under-limit edge case. - Keeps the cap as an over-fetch ceiling without allowing it to make recall narrower than legacy behavior. ## Context Follow-up to PR #205. The original over-fetch implementation used `min(per_query_limit * RECALL_VECTOR_OVERFETCH, RECALL_VECTOR_FETCH_CAP)`, which can ask Qdrant for fewer candidates than the requested response limit when the cap is set low. ## Tests - `venv/bin/pytest tests/test_recall_overfetch.py tests/test_recall_artifact_exclusion.py` - `make test` - A/B already run locally while evaluating PR #205: - `locomo-mini`: main `82.98% (195/235)` -> PR `84.26% (198/235)` - `locomo`: main `81.58% (1258/1542)` -> PR `81.84% (1262/1542)` ## Risk Notes - This is a narrow guardrail around the new config; it should only affect deployments that lower `RECALL_VECTOR_FETCH_CAP` below requested `limit`. - No merge performed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8abbb2f96d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 860e0e7bd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31121fa10f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d18d5d7f30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 participant
Problem
A Claude Desktop session opened with a memory recall that was "basically useless" — it returned 22 memories about Lennard/Berlin but none were the context the user needed ("Lennard's situation we resolved"), so the assistant concluded it had no shared history.
Reproduced against the production corpus (not a local config glitch):
ee830782"Lennard BAföG Mahnung Apr 2026" (919 chars, importance 0.8, tagsbafoeg/mahnung/freistellung/stundung/legal/german-bureaucracy)."...Jobcenter social welfare resolved problem"→ it surfaces at rank Using one server for multiple projects? #4."...Berlin flatmate Bürgergeld Wohngeld"→ it is absent from all 22 results, even though it's inside the time window and the result count (22) was under the limit (25). It never entered the candidate pool.match_type:keywordon the bare token "berlin"; and three importance-0.000MetaPatternartifacts leaked in.Root cause
vector_fetch_limit = per_query_limit). The richer final score (importance, exact-match, tags) only re-ranks memories that already survived that cut — so a high-importance, exact-topic memory that's a slightly weaker pure-vector match is discarded before its signal can lift it. A high-frequency token like "Berlin" saturates the pool and evicts the relevant memory.MetaPatternconsolidation artifacts (importance 0.0, "cluster over 0 days") had no recall exclusion and leaked in via bare-token keyword matches.0.16.0 didn't create these mechanisms; its ranking release amplified them.
Fix
automem/api/recall.py):vector_fetch_limit = min(per_query_limit * RECALL_VECTOR_OVERFETCH, RECALL_VECTOR_FETCH_CAP). The existing re-rank (compute_metadata_score→ sort) then surfaces the right memory — no scoring-weight changes. Results are still trimmed to the requested limit (_guarantee_priority_results), so response size is unchanged.RECALL_VECTOR_OVERFETCH=1restores legacy behavior.automem/search/runtime_recall_helpers.py): excludeRECALL_EXCLUDED_TYPES(default{"MetaPattern"}) at the universal_result_passes_filterschokepoint (covers vector, keyword, metadata, expansion, and state-filter paths) plus the graph/metadata keyword Cypher so artifacts don't consume candidate slots.New config (
automem/config.py):RECALL_VECTOR_OVERFETCH=4,RECALL_VECTOR_FETCH_CAP=200,RECALL_EXCLUDED_TYPES=MetaPattern— all env-overridable.Tests (RED → GREEN)
tests/test_recall_overfetch.py— drives/recallwith a fake Qdrant: asserts the widened fetch limit (20 vs 5), that the target ranked past 1× is recalled and re-ranked into the top results, that the response stays ≤ limit, and an off-switch (OVERFETCH=1) that reproduces the miss.tests/test_recall_artifact_exclusion.py—_result_passes_filtersexcludesMetaPattern/keeps normal types;_graph_keyword_searchemits the Cypher exclusion predicate + param.black+flake8clean.Pre-merge validation (infra/deploy-gated — not run in this PR)
Candidate-pool sizing changes ranking, so gate the merge on the existing harnesses (benchmark ownership stays in
automem):make lab-clone→make lab-queries→make lab-test CONFIG=baseline→make lab-compare CONFIG=fix_v1 BASELINE=baseline.make lab-sweep PARAM=RECALL_VECTOR_OVERFETCH VALUES=1,2,4,8(latency vs recall@k — Qdrant fetch cost scales with the multiplier).make test-locomo/make test-locomo-live.ee830782appears andMetaPatternrows are gone.Out of scope (follow-ups)
entity:organizations:lennard). Needs an enrichment-validator fix + backfill migration.RECALL_RELEVANCE_GATEor surfacing the top evidence score.🤖 Generated with Claude Code