feat(api): enrich refresh_mental_model result_metadata with semantic outcome#2627
Open
hb-cam wants to merge 1 commit into
Open
feat(api): enrich refresh_mental_model result_metadata with semantic outcome#2627hb-cam wants to merge 1 commit into
hb-cam wants to merge 1 commit into
Conversation
…outcome (vectorize-io#2605) refresh_mental_model operations completed with result_metadata carrying only the submit-time {mental_model_id, name} stub — set before the op ran and never enriched — so a monitoring layer could not distinguish "refreshed with real content" from "refreshed empty" without a follow-up content fetch. Retain operations have carried machine-readable outcome metadata since 0.8.x. Mirror the retain pattern: the worker handler now merges the semantic outcome into result_metadata at completion (jsonb ||, preserving the submit-time keys consumers join on): - content_len: length of the final stored content - populated_content: true only for real synthesis — the "No answer provided." reflect fallback and the "Generating content..." placeholder complete wire-successful but read as false (a bare length check would miss them) - based_on_counts: per-fact-type grounding counts from the reflect response The reflect agent's fallback literal is promoted to NO_ANSWER_TEXT so the populated judgment compares against the constant, not a copied string. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
I checked this locally on macOS with Python 3.11.14 and the behavior looks good to me. What I ran:
Result: The extra no-answer stub coverage here is useful, so I am going to close my overlapping #2621 and leave this as the stronger path for #2605. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2605.
What
refresh_mental_modeloperations completed withresult_metadatacarrying only the submit-time{mental_model_id, name}stub (set insubmit_async_refresh_mental_modelbefore the op runs, never enriched). This PR mirrors the retain enrichment pattern (operation_metadata.py+ completion-time metadata write) as suggested in the issue thread: the worker handler now merges the semantic outcome intoresult_metadataat completion.New fields on completed refresh ops:
content_len— length of the final stored contentpopulated_content—trueonly for real synthesis: the reflect agent's"No answer provided."fallback and the"Generating content..."placeholder complete wire-successful but read asfalse(a bare length check would report the 19-char stub as populated)based_on_counts— per-fact-type grounding counts from the reflect responseHow
RefreshMentalModelOutcomeMetadatadataclass alongsideRetainOutcomeMetadatainoperation_metadata.py_write_refresh_outcome_metadatain the engine, mirroring_write_retain_outcome_metadata: jsonb merge (COALESCE(result_metadata, '{}'::jsonb) || …) so the submit-time keys consumers join on are preserved, not replaced; best-effort with a loud warning on failure_handle_refresh_mental_modelafter a successful refresh, before the worker marks the operation completed (same ordering as retain)NO_ANSWER_TEXTconstant so the populated judgment compares against the constant rather than a copied stringNo endpoint or request model changes —
result_metadatais already a free-form dict in the operation status response, so no OpenAPI/client regeneration is needed.Testing
tests/test_refresh_outcome_metadata.py(2 tests, written first and watched fail): a completed refresh exposes all three fields with the submit-time keys intact; the no-answer stub completes but readspopulated_content: falseget_operation_status) against the embedded pg0 database with only the LLM-boundrefresh_mental_modelstubbedtest_mental_model_scheduled_refresh.py,test_mental_model_delta.py,test_mental_model_hooks.py(24 passed, 4 skipped),test_async_batch_retain.py(22 passed)ruff check,ruff format --check,ty checkclean on changed files🤖 Generated with Claude Code