Fix user-scoped memory consolidation#1112
Conversation
lightzt99
left a comment
There was a problem hiding this comment.
Thanks for this fix. CI is green and the sync/async + SQLite/OceanBase + nested AND/OR test coverage is solid.
Before merging, please consider the following:
P1 — Two fail-open paths conflict with this PR's goal (scope isolation)
$ninsilently passes all rows whenop_valueis not a list/tuple/set._filter_value_matchesonly checksisinstance(op_value, (list, tuple, set)) and actual in op_value → return False; whenisinstancefails it falls through to the trailingreturn True.$inis fail-closed (not isinstance(...) or actual not in op_value → return False);$ninshould be symmetric._memory_matches_filter_expression:if not isinstance(filters, dict): return True. A stray string/number passed asfilterswould let every memory through. Shouldreturn False(fail-closed) or raise.
P1 — search_memories overscan cap returns fewer than limit for selective filters
search_limit = max(limit * 5, 100). With 10k rows, 1% selectivity (100 matches), limit=10: overscan 100, Python filters to 1, return 1 — but 100 actually match and the user asked for 10. get_all_memories uses limit=None (full scan) so it doesn't have this bug, but search_memories shouldn't under-return.
P2 — Unknown filter operators silently drop all rows
_filter_value_matches returns False for $regex / $exists / $between / $all. Users get an empty result with no warning, believing the filter worked. Suggest raise ValueError(f"Unsupported filter operator: {op}").
P2 — like / $like anchor semantics lost
str(op_value).replace("%", "") substring match collapses like "private%" (prefix), like "%private" (suffix), and like "%private%" (contains) to the same contains behavior. SQL LIKE anchors silently don't work — risky for permission/scope filters. Either implement SQL semantics or document "substring contains" only.
P2 — DELETE rejection path untested
Tests cover UPDATE rejection (sync + async), but no test verifies that _intelligent_add correctly omits the result when delete is rejected by scope. Add a sync + async DELETE-rejected case.
P3 — count_all_memories full-table scan has no upper bound
filters_fully_pushed=False → len(get_all_memories(limit=None)) pulls every matching row into Python. Dashboard calls count to show totals — at 100k memories a single count is O(N). Suggest a cap (e.g. 100k) or document the limitation.
P3 — Warning message is inaccurate
_intelligent_add logs "Skipped UPDATE for memory %s because it is outside the requested scope", but _update_memory returning None may also mean the memory was concurrently deleted or never existed. Suggest "not found or outside requested scope".
P3 — Single 740-line commit is hard to review/rollback
Three independent changes (_intelligent_add truthy return / search-list defensive filter / OceanBase logical filter translation) are bundled. Splitting into 3 commits would help.
Band-aid nature
The defensive re-check at the search/list tail is a patch, not a root-cause fix. If the real cause is SQLite list() not pushing user_id filters, the next new store path will bypass this defense too. Worth a follow-up to fix the backend's missing filter; keep this layer as defense-in-depth.
Summary: direction is right; please fix the two P1 fail-open paths and the overscan cap before merge, the rest can be follow-ups.
cc38788 to
32f6b95
Compare
Summary
Validation
python -m pytest tests/unit/test_issue_1109_user_scope.py tests/unit/test_list_memory_filters.py tests/unit/test_oceanbase_native_filter.py -qpython -m pytest tests/integration/test_noop_embedding_mode.py tests/integration/test_storage_integration.py tests/integration/test_async_memory_integration.py -qCloses #1109.