Skip to content

feat(api): enrich refresh_mental_model result_metadata with semantic outcome#2627

Open
hb-cam wants to merge 1 commit into
vectorize-io:mainfrom
hb-cam:refresh-outcome-metadata-2605
Open

feat(api): enrich refresh_mental_model result_metadata with semantic outcome#2627
hb-cam wants to merge 1 commit into
vectorize-io:mainfrom
hb-cam:refresh-outcome-metadata-2605

Conversation

@hb-cam

@hb-cam hb-cam commented Jul 8, 2026

Copy link
Copy Markdown

Fixes #2605.

What

refresh_mental_model operations completed with result_metadata carrying only the submit-time {mental_model_id, name} stub (set in submit_async_refresh_mental_model before the op runs, never enriched). This PR mirrors the retain enrichment pattern (operation_metadata.py + completion-time metadata write) as suggested in the issue thread: the worker handler now merges the semantic outcome into result_metadata at completion.

New fields on completed refresh ops:

  • content_len — length of the final stored content
  • populated_contenttrue only for real synthesis: the reflect agent's "No answer provided." fallback and the "Generating content..." placeholder complete wire-successful but read as false (a bare length check would report the 19-char stub as populated)
  • based_on_counts — per-fact-type grounding counts from the reflect response

How

  • RefreshMentalModelOutcomeMetadata dataclass alongside RetainOutcomeMetadata in operation_metadata.py
  • _write_refresh_outcome_metadata in the engine, mirroring _write_retain_outcome_metadata: jsonb merge (COALESCE(result_metadata, '{}'::jsonb) || …) so the submit-time keys consumers join on are preserved, not replaced; best-effort with a loud warning on failure
  • Called from _handle_refresh_mental_model after a successful refresh, before the worker marks the operation completed (same ordering as retain)
  • The reflect agent's fallback literal is promoted to a NO_ANSWER_TEXT constant so the populated judgment compares against the constant rather than a copied string

No endpoint or request model changes — result_metadata is already a free-form dict in the operation status response, so no OpenAPI/client regeneration is needed.

Testing

  • New tests/test_refresh_outcome_metadata.py (2 tests, written first and watched fail): a completed refresh exposes all three fields with the submit-time keys intact; the no-answer stub completes but reads populated_content: false
  • Tests exercise the real path (submit → task backend → handler → metadata write → get_operation_status) against the embedded pg0 database with only the LLM-bound refresh_mental_model stubbed
  • Neighboring suites pass: test_mental_model_scheduled_refresh.py, test_mental_model_delta.py, test_mental_model_hooks.py (24 passed, 4 skipped), test_async_batch_retain.py (22 passed)
  • ruff check, ruff format --check, ty check clean on changed files

🤖 Generated with Claude Code

…outcome (vectorize-io#2605)

refresh_mental_model operations completed with result_metadata carrying only
the submit-time {mental_model_id, name} stub — set before the op ran and never
enriched — so a monitoring layer could not distinguish "refreshed with real
content" from "refreshed empty" without a follow-up content fetch. Retain
operations have carried machine-readable outcome metadata since 0.8.x.

Mirror the retain pattern: the worker handler now merges the semantic outcome
into result_metadata at completion (jsonb ||, preserving the submit-time keys
consumers join on):

- content_len: length of the final stored content
- populated_content: true only for real synthesis — the "No answer provided."
  reflect fallback and the "Generating content..." placeholder complete
  wire-successful but read as false (a bare length check would miss them)
- based_on_counts: per-fact-type grounding counts from the reflect response

The reflect agent's fallback literal is promoted to NO_ANSWER_TEXT so the
populated judgment compares against the constant, not a copied string.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@koriyoshi2041

Copy link
Copy Markdown
Contributor

I checked this locally on macOS with Python 3.11.14 and the behavior looks good to me.

What I ran:

  • git diff --check origin/main...HEAD
  • env -u ALL_PROXY -u all_proxy -u HTTPS_PROXY -u https_proxy -u HTTP_PROXY -u http_proxy uv run pytest tests/test_refresh_outcome_metadata.py tests/test_mental_model_scheduled_refresh.py -q -n0

Result: 8 passed, 2 warnings in 21.83s. The first run inherited my local SOCKS proxy env and failed before exercising the patch because socksio is not installed; rerunning without proxy env cleared that and the focused refresh tests passed.

The extra no-answer stub coverage here is useful, so I am going to close my overlapping #2621 and leave this as the stronger path for #2605.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refresh_mental_model result_metadata lacks semantic outcome fields (retain ops have them since 0.8.x)

2 participants