fix(merge): surgical integration of upstream volcengine#1159 — 4 files

The initial merge of upstream 'Feature/memory opt' (volcengine#1159) auto-merged a
unified operations schema into utils/uri.py and utils/content.py while our
TAKE DEV choice left memory_updater.py / extract_loop.py expecting the
old split schema. Runtime smoke test revealed cascading API mismatches:

  - 'ResolvedOperations' has no 'write_operations' (AttributeError)
  - serialize_with_metadata() arg order mismatch (TypeError on write)
  - LLM output validation would fail on missing reasoning/edit_overview_uris
    fields (upstream commented them out in schema_model_generator)

Two parallel analysis agents (arch map of volcengine#1159 + call-graph audit) both
converged on: TAKE DEV for schema-breaking changes, KEEP upstream for
purely-additive improvements. This commit is the resulting minimal diff.

Reverted to dev (schema-breaking bits):
- utils/uri.py: ResolvedOperations keeps write_operations/edit_operations
  split (load-bearing for brain-hardening _MERGE_ON_COLLISION_TYPES)
- utils/content.py: serialize_with_metadata(content, metadata) positional
  signature matches all 5 call sites (memory_updater x2, memory_type_registry
  x1, content_write x2 — upstream's new content_write also uses this form)
- schema_model_generator.py: uncommented 'reasoning' and 'edit_overview_uris'
  Pydantic field definitions — extract_loop.py:196 requires both

Patched to bridge upstream additive + dev signature:
- memory_type_registry.py::initialize_memory_files — render content_template
  inline with jinja2, then call dev's serialize_with_metadata(rendered,
  metadata) instead of upstream's serialize_with_metadata(metadata,
  content_template=, extract_context=)

Kept upstream wins untouched (already in merged state, fully additive):
- MemoryTypeRegistry.initialize_memory_files() method (volcengine#1159)
- MemoryTypeRegistry.__init__(load_schemas=True) auto-loading
- ExtractContext.get_year/get_month/get_day helpers (used by events.yaml)
- openviking/telemetry/tracer.py module
- compressor_v2 v2_lock_retry config + tracer (volcengine#1275)
- merge_op/base.py get_first_replace() helper
- merge_op/patch.py None-handling branch
- MemoryField.init_value
- soul.yaml / identity.yaml prompt templates
- events.yaml year/month/day folder structure
- AGFS Rust binding (volcengine#1221) — confirmed active at runtime

Smoke test result (conv-26 session_1, qwopus-4b, temp=0.1):
  Written: 3, Edited: 0, Deleted: 0, Errors: 0
  4 memories total (2 entities, 1 episode, 1 other)
  Historical baseline variance range: 4-10 memories, median ~6.
  Result within normal LLM variance.

Co-Authored-By: claude-flow <ruv@ruv.net>

Author: nvidia

openviking/session/memory/memory_type_registry.py

@@ -252,19 +252,26 @@ async def initialize_memory_files(self, ctx: Any) -> None:
             except Exception:
                 pass
 
-            # Add MEMORY_FIELDS comment with field metadata
-            # Template rendering is handled inside serialize_with_metadata
+            # Render content_template inline (dev's serialize_with_metadata takes
+            # content as a positional arg and does not do template rendering itself).
             from openviking.session.memory.utils.content import serialize_with_metadata
 
+            try:
+                rendered_content = env.from_string(schema.content_template).render(
+                    **fields_with_init
+                )
+            except Exception as e:
+                logger.warning(
+                    f"[MemoryTypeRegistry] Failed to render content template for "
+                    f"{schema.memory_type}: {e}"
+                )
+                rendered_content = ""
+
             metadata = {
                 "memory_type": schema.memory_type,
                 **fields_with_init,
-                "content": "",  # content will come from content_template rendering
             }
-            full_content = serialize_with_metadata(
-                metadata,
-                content_template=schema.content_template,
-            )
+            full_content = serialize_with_metadata(rendered_content, metadata)
 
             # Write the file
             try:

openviking/session/memory/schema_model_generator.py

@@ -242,10 +242,10 @@ def create_structured_operations_model(self) -> Type[BaseModel]:
         # Build field definitions for each memory_type
         field_definitions: Dict[str, Tuple[Type[Any], Any]] = {}
 
-        # field_definitions["reasoning"] = (
-        #     str,
-        #     Field("", description="reasoning"),
-        # )
+        field_definitions["reasoning"] = (
+            str,
+            Field("", description="reasoning"),
+        )
 
         for mt in enabled_memory_types:
             flat_model = self.create_flat_data_model(mt)
@@ -267,17 +267,17 @@ def create_structured_operations_model(self) -> Type[BaseModel]:
                 )
 
         # Use single generic model for overview edit (same for all memory types)
-        # generic_overview_edit = self.create_overview_edit_model(
-        #     enabled_memory_types[0] if enabled_memory_types else None
-        # )
-
-        # field_definitions["edit_overview_uris"] = (
-        #     List[generic_overview_edit],  # type: ignore
-        #     Field(
-        #         default_factory=list,
-        #         description="Edit operations for .overview.md files using memory_type",
-        #     ),
-        # )
+        generic_overview_edit = self.create_overview_edit_model(
+            enabled_memory_types[0] if enabled_memory_types else None
+        )
+
+        field_definitions["edit_overview_uris"] = (
+            List[generic_overview_edit],  # type: ignore
+            Field(
+                default_factory=list,
+                description="Edit operations for .overview.md files using memory_type",
+            ),
+        )
 
         field_definitions["delete_uris"] = (
             List[str],
@@ -330,7 +330,7 @@ def to_legacy_operations(self) -> Dict[str, Any]:
             return {
                 "write_uris": write_uris,
                 "edit_uris": edit_uris,
-                #"edit_overview_uris": self.edit_overview_uris,
+                "edit_overview_uris": self.edit_overview_uris,
                 "delete_uris": self.delete_uris,
             }
 

openviking/session/memory/utils/content.py

@@ -39,47 +39,27 @@ def _deserialize_datetime(metadata: Dict[str, Any]) -> Dict[str, Any]:
     return result
 
 
-def serialize_with_metadata(
-    metadata: Dict[str, Any],
-    content_template: str = None,
-    extract_context: Any = None,
-) -> str:
+def serialize_with_metadata(content: str, metadata: Dict[str, Any]) -> str:
     """
     Serialize content and metadata into a single string.
 
     The metadata is stored in an HTML comment at the end of the content.
 
     Args:
-        metadata: Dictionary containing all fields including "content".
-                  content is extracted and used as the main body.
-        content_template: Optional Jinja2 template to render content.
-        extract_context: Optional context for template rendering.
+        content: The main memory content (Markdown)
+        metadata: Dictionary containing metadata fields:
+            - memory_type: Type of memory (NOT included in output)
+            - fields: Structured fields (for template mode)
+            - name: Memory name
+            - tags: List of tags
+            - created_at: Creation datetime
+            - updated_at: Update datetime
+            - abstract: L0 abstract
+            - overview: L1 overview
 
     Returns:
         Combined string with content followed by metadata in HTML comment
     """
-    # Extract content from metadata (default to empty string)
-    content = metadata.pop("content", "") or ""
-
-    # Render template if provided
-    if content_template:
-        try:
-            import jinja2
-            from jinja2 import Environment
-
-            env = Environment(autoescape=False, undefined=jinja2.DebugUndefined)
-            template_vars = metadata.copy()
-            template_vars["extract_context"] = extract_context
-
-            jinja_template = env.from_string(content_template)
-            content = jinja_template.render(**template_vars).strip()
-        except Exception:
-            # If template rendering fails, use content as-is
-            pass
-
-    # Restore metadata (we popped content earlier)
-    # Note: metadata dict is modified in place, caller should be aware
-
     # Clean metadata - remove None values and memory_type
     clean_metadata = {k: v for k, v in metadata.items() if v is not None and k != "memory_type"}
 

openviking/session/memory/utils/uri.py

@@ -159,12 +159,12 @@ def generate_uri(
     if not uri_template:
         raise ValueError("Memory type has neither directory nor filename_template")
 
-    # Build the context for Jinja2 rendering - include user_space and agent_space
+    # Build the context for Jinja2 rendering
     context = {
         "user_space": user_space,
         "agent_space": agent_space,
     }
-    # Add all fields to context (uri_fields with actual values)
+    # Add all fields to context
     context.update(fields)
 
     # Render using unified render_template method (same as content_template)
@@ -336,7 +336,6 @@ def is_uri_allowed_for_schema(
     schemas: List[MemoryTypeSchema],
     user_space: str = "default",
     agent_space: str = "default",
-    extract_context: Any = None,
 ) -> bool:
     """
     Check if a URI is allowed for the given activated schemas.
@@ -346,15 +345,12 @@ def is_uri_allowed_for_schema(
         schemas: List of activated memory type schemas
         user_space: User space to substitute for {{ user_space }}
         agent_space: Agent space to substitute for {{ agent_space }}
-        extract_context: ExtractContext instance for template rendering
 
     Returns:
         True if the URI is allowed
     """
     allowed_dirs = collect_allowed_directories(schemas, user_space, agent_space, extract_context)
-    allowed_patterns = collect_allowed_path_patterns(
-        schemas, user_space, agent_space, extract_context
-    )
+    allowed_patterns = collect_allowed_path_patterns(schemas, user_space, agent_space, extract_context)
     return is_uri_allowed(uri, allowed_dirs, allowed_patterns)
 
 
@@ -504,8 +500,8 @@ class ResolvedOperations:
     """Operations with resolved URIs."""
 
     def __init__(self):
-        # Unified operations list - all are edit (will read existing file first)
-        self.operations: List[ResolvedOperation] = []
+        self.write_operations: List[ResolvedOperation] = []
+        self.edit_operations: List[ResolvedOperation] = []
         self.edit_overview_operations: List[
             Tuple[Any, str]
         ] = []  # (overview_edit_model, overview_uri)
@@ -526,9 +522,7 @@ def resolve_all_operations(
     """
     Resolve URIs for all operations.
 
-    Uses per-memory_type format (e.g., soul, identity fields).
-    All operations are unified into a single list - each will attempt to read existing
-    file first, then merge (or write new if not exists).
+    Supports both legacy format (write_uris/edit_uris) and new per-memory_type format.
 
     Args:
         operations: StructuredMemoryOperations
@@ -552,32 +546,67 @@ def resolve_all_operations(
                 continue
             items = value if isinstance(value, list) else [value]
             for item in items:
+                # Determine if edit (has uri) or write
+                is_edit = False
+                if hasattr(item, "uri") and item.uri:
+                    is_edit = True
+                elif isinstance(item, dict) and item.get("uri"):
+                    is_edit = True
                 # Convert to dict for URI resolution
                 item_dict = dict(item) if hasattr(item, "model_dump") else dict(item)
                 try:
                     uri = resolve_flat_model_uri(
-                        item_dict,
-                        registry,
-                        user_space,
-                        agent_space,
-                        memory_type=field_name,
-                        extract_context=extract_context,
-                    )
-                    # All operations go to unified list - will read existing file first
-                    resolved.operations.append(
-                        ResolvedOperation(model=item_dict, uri=uri, memory_type=field_name)
+                        item_dict, registry, user_space, agent_space,
+                        memory_type=field_name, extract_context=extract_context
                     )
+                    if is_edit:
+                        resolved.edit_operations.append(
+                            ResolvedOperation(model=item_dict, uri=uri, memory_type=field_name)
+                        )
+                    else:
+                        resolved.write_operations.append(
+                            ResolvedOperation(model=item_dict, uri=uri, memory_type=field_name)
+                        )
                 except Exception as e:
                     resolved.errors.append(f"Failed to resolve {field_name} operation: {e}")
+    else:
+        # Legacy format
+        write_uris = operations.write_uris if hasattr(operations, "write_uris") else []
+        edit_uris = operations.edit_uris if hasattr(operations, "edit_uris") else []
+
+        for op in write_uris:
+            try:
+                uri = resolve_flat_model_uri(
+                    op, registry, user_space, agent_space, extract_context=extract_context
+                )
+                # Legacy format: try to get memory_type from model, otherwise empty
+                memory_type = op.get("memory_type", "") if isinstance(op, dict) else ""
+                resolved.write_operations.append(
+                    ResolvedOperation(model=op, uri=uri, memory_type=memory_type)
+                )
+            except Exception as e:
+                resolved.errors.append(f"Failed to resolve write operation: {e}")
+
+        for op in edit_uris:
+            try:
+                uri = resolve_flat_model_uri(
+                    op, registry, user_space, agent_space, extract_context=extract_context
+                )
+                memory_type = op.get("memory_type", "") if isinstance(op, dict) else ""
+                resolved.edit_operations.append(
+                    ResolvedOperation(model=op, uri=uri, memory_type=memory_type)
+                )
+            except Exception as e:
+                resolved.errors.append(f"Failed to resolve edit operation: {e}")
 
     # Resolve edit_overview operations (overview edit models)
-    # if hasattr(operations, "edit_overview_uris"):
-    #     for op in operations.edit_overview_uris:
-    #         try:
-    #             uri = resolve_overview_edit_uri(op, registry, user_space, agent_space)
-    #             resolved.edit_overview_operations.append((op, uri))
-    #         except Exception as e:
-    #             resolved.errors.append(f"Failed to resolve edit_overview operation: {e}")
+    if hasattr(operations, "edit_overview_uris"):
+        for op in operations.edit_overview_uris:
+            try:
+                uri = resolve_overview_edit_uri(op, registry, user_space, agent_space)
+                resolved.edit_overview_operations.append((op, uri))
+            except Exception as e:
+                resolved.errors.append(f"Failed to resolve edit_overview operation: {e}")
 
     # Resolve delete operations (already URI strings)
     if hasattr(operations, "delete_uris"):
@@ -614,24 +643,24 @@ def validate_operations_uris(
         Tuple of (is_valid, list of error messages)
     """
     allowed_dirs = collect_allowed_directories(schemas, user_space, agent_space, extract_context)
-    allowed_patterns = collect_allowed_path_patterns(
-        schemas, user_space, agent_space, extract_context
-    )
+    allowed_patterns = collect_allowed_path_patterns(schemas, user_space, agent_space, extract_context)
 
     errors = []
 
     # First resolve all URIs
-    resolved = resolve_all_operations(
-        operations, registry, user_space, agent_space, extract_context
-    )
+    resolved = resolve_all_operations(operations, registry, user_space, agent_space, extract_context)
 
     if resolved.has_errors():
         errors.extend(resolved.errors)
     else:
-        # Validate resolved URIs - all operations use unified list
-        for resolved_op in resolved.operations:
+        # Validate resolved URIs
+        for resolved_op in resolved.write_operations:
+            if not is_uri_allowed(resolved_op.uri, allowed_dirs, allowed_patterns):
+                errors.append(f"Write operation URI not allowed: {resolved_op.uri}")
+
+        for resolved_op in resolved.edit_operations:
             if not is_uri_allowed(resolved_op.uri, allowed_dirs, allowed_patterns):
-                errors.append(f"Operation URI not allowed: {resolved_op.uri}")
+                errors.append(f"Edit operation URI not allowed: {resolved_op.uri}")
 
         for _op, uri in resolved.edit_overview_operations:
             if not is_uri_allowed(uri, allowed_dirs, allowed_patterns):