improvements to tagging, filtering

Author: Aayush Kataria

Docs/architecture/orchestrators.md

@@ -0,0 +1,23 @@
+# Durable orchestrator activity chains
+
+W15 splits LLM extraction from persistence so Durable retries after a Cosmos write failure do not re-run the LLM activity.
+
+```text
+ExtractMemoriesOrchestrator
+  em_Extract  (load recent turns + LLM + parse; no embeddings/writes)
+  em_Persist  (embeddings + deterministic create_item; 409 = already persisted)
+  em_ReconcileMemories (optional; single activity for GA)
+
+ThreadSummaryOrchestrator
+  ts_Extract
+  ts_PersistSummary
+
+UserSummaryOrchestrator
+  us_Extract
+  us_PersistUserSummary
+
+SynthesizeProceduralOrchestrator
+  sp_SynthesizeProcedural (single activity for GA)
+```
+
+Fact and episodic IDs are deterministic from user, thread, and normalized content. Thread and user summaries keep their deterministic summary IDs.

Docs/operations.md

@@ -0,0 +1,18 @@
+# Operations
+
+## Memory lifecycle (TTL)
+
+| Type | Default TTL | Source |
+|---|---:|---|
+| turn | 30 d | container default (memories_turns) |
+| episodic | 90 d | per-doc ttl (memories container) |
+| thread_summary | never | container default (memories, -1) |
+| user_summary | never | container default |
+| fact | never | container default; supersession handles aging |
+| procedural | never | container default; supersession handles aging |
+
+Override per write:
+    client.add_memory(text, type="turn", ttl=60)   # expires in 60 seconds
+
+Override per container at provision time:
+    azd env set MEMORIES_TURNS_DEFAULT_TTL 86400   # 1 day

Docs/public_api.md

@@ -27,15 +27,15 @@
 - `delete_local(memory_id) -> None` — remove a local buffered memory.
 - `add_cosmos(user_id, role, content, memory_type='turn', metadata=None, thread_id=None, tags=None, ttl=None, salience=None, embedding=None, embed=None) -> str` — upsert one memory to Cosmos and return its id.
 - `push_to_cosmos(batch_size=25) -> None` — flush local buffered memories to Cosmos.
-- `get_memories(memory_id=None, user_id=None, thread_id=None, role=None, memory_types=None, recent_k=None, tags=None, any_tags=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None) -> list[dict]` — retrieve memories with filters.
+- `get_memories(memory_id=None, user_id=None, thread_id=None, role=None, memory_types=None, recent_k=None, tags_all=None, tags_any=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None, created_after=None, created_before=None) -> list[dict]` — retrieve memories with filters.
 - `update_cosmos(memory_id, content=None, role=None, memory_type=None, metadata=None) -> None` — update a Cosmos memory.
 - `delete_cosmos(memory_id, thread_id, user_id) -> None` — delete a Cosmos memory.
-- `get_thread(thread_id, user_id=None, memory_types=None, recent_k=None, tags=None, exclude_tags=None, include_superseded=False) -> list[dict]` — retrieve a thread oldest-first.
+- `get_thread(thread_id, user_id=None, memory_types=None, recent_k=None, tags_all=None, tags_any=None, exclude_tags=None, include_superseded=False, created_after=None, created_before=None) -> list[dict]` — retrieve a thread oldest-first.
 - `get_user_summary(user_id) -> Optional[dict]` — retrieve the active user-summary document.
 
 ### Retrieval
 
-- `search_cosmos(search_terms, memory_id=None, user_id=None, role=None, memory_types=None, thread_id=None, hybrid_search=False, top_k=5, tags=None, any_tags=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None) -> list[dict]` — vector or hybrid search memories.
+- `search_cosmos(search_terms, memory_id=None, user_id=None, role=None, memory_types=None, thread_id=None, hybrid_search=False, top_k=5, tags_all=None, tags_any=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None, created_after=None, created_before=None) -> list[dict]` — vector or hybrid search memories.
 - `get_procedural_prompt(user_id) -> Optional[str]` — read the active procedural prompt.
 - `get_procedural_history(user_id, limit=10) -> list[dict]` — read procedural prompt history.
 - `get_procedural_memories(user_id, priority=None, category=None, min_salience=None, include_superseded=False) -> list[dict]` — retrieve procedural memory documents.
@@ -57,6 +57,7 @@
 
 - `add_tags(memory_id, user_id, thread_id, tags) -> None` — add tags to a memory.
 - `remove_tags(memory_id, user_id, thread_id, tags) -> None` — remove tags from a memory.
+- `list_tags(user_id, *, thread_id=None, prefix=None, include_sys=False) -> list[str]` — list sorted, deduped tags for a user; omits `sys:*` by default.
 
 ## AsyncCosmosMemoryClient
 
@@ -77,15 +78,15 @@ Local-buffer methods remain synchronous in-memory operations; Cosmos, retrieval,
 - `delete_local(memory_id) -> None` — remove a local buffered memory.
 - `async add_cosmos(user_id, role, content, memory_type='turn', metadata=None, thread_id=None, tags=None, ttl=None, salience=None, embedding=None, embed=None) -> str` — upsert one memory to Cosmos and return its id.
 - `async push_to_cosmos(batch_size=25) -> None` — flush local buffered memories to Cosmos.
-- `async get_memories(memory_id=None, user_id=None, thread_id=None, role=None, memory_types=None, recent_k=None, tags=None, any_tags=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None) -> list[dict]` — retrieve memories with filters.
+- `async get_memories(memory_id=None, user_id=None, thread_id=None, role=None, memory_types=None, recent_k=None, tags_all=None, tags_any=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None, created_after=None, created_before=None) -> list[dict]` — retrieve memories with filters.
 - `async update_cosmos(memory_id, content=None, role=None, memory_type=None, metadata=None) -> None` — update a Cosmos memory.
 - `async delete_cosmos(memory_id, thread_id, user_id) -> None` — delete a Cosmos memory.
-- `async get_thread(thread_id, user_id=None, memory_types=None, recent_k=None, tags=None, exclude_tags=None, include_superseded=False) -> list[dict]` — retrieve a thread oldest-first.
+- `async get_thread(thread_id, user_id=None, memory_types=None, recent_k=None, tags_all=None, tags_any=None, exclude_tags=None, include_superseded=False, created_after=None, created_before=None) -> list[dict]` — retrieve a thread oldest-first.
 - `async get_user_summary(user_id) -> Optional[dict]` — retrieve the active user-summary document.
 
 ### Retrieval
 
-- `async search_cosmos(search_terms, memory_id=None, user_id=None, role=None, memory_types=None, thread_id=None, hybrid_search=False, top_k=5, tags=None, any_tags=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None) -> list[dict]` — vector or hybrid search memories.
+- `async search_cosmos(search_terms, memory_id=None, user_id=None, role=None, memory_types=None, thread_id=None, hybrid_search=False, top_k=5, tags_all=None, tags_any=None, exclude_tags=None, include_superseded=False, min_salience=None, min_confidence=None, created_after=None, created_before=None) -> list[dict]` — vector or hybrid search memories.
 - `async get_procedural_prompt(user_id) -> Optional[str]` — read the active procedural prompt.
 - `async get_procedural_history(user_id, limit=10) -> list[dict]` — read procedural prompt history.
 - `async get_procedural_memories(user_id, priority=None, category=None, min_salience=None, include_superseded=False) -> list[dict]` — retrieve procedural memory documents.
@@ -107,6 +108,7 @@ Local-buffer methods remain synchronous in-memory operations; Cosmos, retrieval,
 
 - `async add_tags(memory_id, user_id, thread_id, tags) -> None` — add tags to a memory.
 - `async remove_tags(memory_id, user_id, thread_id, tags) -> None` — remove tags from a memory.
+- `async list_tags(user_id, *, thread_id=None, prefix=None, include_sys=False) -> list[str]` — list sorted, deduped tags for a user; omits `sys:*` by default.
 
 ## Extension Points
 

Samples/Scenarios/scenario_tagging_and_filtering.py

@@ -4,8 +4,8 @@
 feature. Tags are arbitrary string labels attached to a memory at write
 time; at read time you can filter retrieval with three composable predicates:
 
-* ``tags=[...]``        — AND filter: every listed tag must be present
-* ``any_tags=[...]``    — OR filter: at least one listed tag must be present
+* ``tags_all=[...]``    — AND filter: every listed tag must be present
+* ``tags_any=[...]``    — OR filter: at least one listed tag must be present
 * ``exclude_tags=[...]`` — NOT filter: none of these tags may be present
 
 Tags are stored in Cosmos as a JSON array on the memory document. They work
@@ -90,24 +90,24 @@ def main() -> None:
         print(f"  + {mem_type:>10}  tags={tags}")
 
     # ------------------------------------------------------------------
-    # 2. AND filter: tags=[...] — every listed tag must be present
+    # 2. AND filter: tags_all=[...] — every listed tag must be present
     #    Use get_memories() for filter-only retrieval (no vector search).
     # ------------------------------------------------------------------
-    print_section("AND filter — tags=['workflow', 'important']")
+    print_section("AND filter — tags_all=['workflow', 'important']")
     results = client.get_memories(
         user_id=user_id,
-        tags=["workflow", "important"],
+        tags_all=["workflow", "important"],
     )
     print_results(results)
     # Expected: only the "Always confirm before purging" procedural memory.
 
     # ------------------------------------------------------------------
-    # 3. OR filter: any_tags=[...] — any listed tag may match
+    # 3. OR filter: tags_any=[...] — any listed tag may match
     # ------------------------------------------------------------------
-    print_section("OR filter — any_tags=['diet', 'travel']")
+    print_section("OR filter — tags_any=['diet', 'travel']")
     results = client.get_memories(
         user_id=user_id,
-        any_tags=["diet", "travel"],
+        tags_any=["diet", "travel"],
     )
     print_results(results)
     # Expected: the peanut allergy fact and the PyCon episodic memory.
@@ -129,11 +129,11 @@ def main() -> None:
     #    search_cosmos() runs vector similarity against search_terms and
     #    composes naturally with structural / tag filters.
     # ------------------------------------------------------------------
-    print_section("Semantic + tag filter — search_terms='shipping a service' tags=['work']")
+    print_section("Semantic + tag filter — search_terms='shipping a service' tags_all=['work']")
     results = client.search_cosmos(
         search_terms="shipping a service",
         user_id=user_id,
-        tags=["work"],
+        tags_all=["work"],
         top_k=5,
     )
     print_results(results)
@@ -143,10 +143,10 @@ def main() -> None:
     # ------------------------------------------------------------------
     # 6. Compose AND + NOT filters across types
     # ------------------------------------------------------------------
-    print_section("AND + NOT — tags=['preference']  exclude_tags=['health']")
+    print_section("AND + NOT — tags_all=['preference']  exclude_tags=['health']")
     results = client.get_memories(
         user_id=user_id,
-        tags=["preference"],
+        tags_all=["preference"],
         exclude_tags=["health"],
     )
     print_results(results)

agent_memory_toolkit/__init__.py

@@ -9,6 +9,7 @@
     CosmosNotConnectedError,
     CosmosOperationError,
     LLMError,
+    MemoryConflictError,
     MemoryNotFoundError,
     ValidationError,
 )
@@ -50,6 +51,7 @@
     "CosmosNotConnectedError",
     "CosmosOperationError",
     "LLMError",
+    "MemoryConflictError",
     "MemoryNotFoundError",
     "ValidationError",
     "DEFAULT_FACT_EXTRACTION_EVERY_N",

agent_memory_toolkit/_query_builder.py

@@ -77,6 +77,31 @@ def add_gte(self, field: str, param_name: str, value: Any) -> None:
         self._conditions.append(f"{field} >= {param_name}")
         self._parameters.append({"name": param_name, "value": value})
 
+    def add_time_range(
+        self,
+        field: str,
+        *,
+        after: Any = None,
+        before: Any = None,
+        after_param: str = "@after",
+        before_param: str = "@before",
+    ) -> None:
+        """Add inclusive lower/upper bounds for an ISO-sortable time field."""
+        if after is not None:
+            self._conditions.append(f"{field} >= {after_param}")
+            self._parameters.append({"name": after_param, "value": after})
+        if before is not None:
+            self._conditions.append(f"{field} <= {before_param}")
+            self._parameters.append({"name": before_param, "value": before})
+
+    def add_metadata_filter(self, path: str, op: str, value: Any, *, param_name: str | None = None) -> None:
+        """Add a parameterized comparison filter for a metadata path."""
+        if op not in {"=", "!=", ">", "<", ">=", "<="}:
+            raise ValueError(f"unsupported op: {op}")
+        pname = param_name or f"@m_{len(self._parameters)}"
+        self._conditions.append(f"{path} {op} {pname}")
+        self._parameters.append({"name": pname, "value": value})
+
     def build_where(self) -> str:
         """Return the ``WHERE …`` clause (or empty string if no filters)."""
         if not self._conditions:

agent_memory_toolkit/_utils.py

@@ -15,6 +15,7 @@
 
 from ._query_builder import _QueryBuilder
 from .exceptions import ConfigurationError, ValidationError
+from .thresholds import DEFAULT_TTL_BY_TYPE as DEFAULT_TTL_BY_TYPE, default_ttl_for
 
 VALID_ROLES = {"agent", "user", "tool", "system"}
 VALID_TYPES = {"turn", "summary", "fact", "user_summary", "procedural", "episodic"}
@@ -57,17 +58,14 @@ def new_user_summary_id() -> str:
     """Return a fresh ``user_summary_*`` id."""
     return new_id("user_summary")
 
-DEFAULT_TTL_BY_TYPE: dict[str, int | None] = {
-    "turn": 2_592_000,  # 30 days
-    "summary": None,
-    "fact": None,
-    "user_summary": None,
-    "procedural": None,
-    "episodic": 7_776_000,  # 90 days
-}
+_WHITESPACE_RE = re.compile(r"\s+")
 
 
-_WHITESPACE_RE = re.compile(r"\s+")
+def _coerce_datetime_iso(value: Optional[str | datetime]) -> Optional[str]:
+    """Return ISO text for datetime values while leaving strings unchanged."""
+    if isinstance(value, datetime):
+        return value.isoformat()
+    return value
 
 
 def _normalize_for_hash(text: str) -> str:
@@ -114,7 +112,7 @@ def _make_memory(
         raise ValidationError(f"type must be one of {VALID_TYPES}, got '{memory_type}'")
 
     if ttl is None:
-        ttl = DEFAULT_TTL_BY_TYPE.get(memory_type)
+        ttl = default_ttl_for(memory_type)
 
     memory: dict[str, Any] = {
         "id": memory_id or str(uuid.uuid4()),