Skip to content

refactor(eval): unify evaluation strategy solvers using generic InferenceFormat solver#1983

Merged
gspencergoog merged 0 commit into
unified-inference-formats-strategiesfrom
unified-inference-formats-eval
Jul 13, 2026
Merged

refactor(eval): unify evaluation strategy solvers using generic InferenceFormat solver#1983
gspencergoog merged 0 commit into
unified-inference-formats-strategiesfrom
unified-inference-formats-eval

Conversation

@gspencergoog

@gspencergoog gspencergoog commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Integrates the unified InferenceStrategy and Parser abstractions into the A2UI evaluation framework, replacing redundant solver scripts with a single generic format solver.

Changes

  • Unification of Evaluation Solvers:
    • Deleted duplicate elemental and express solver files (eval/a2ui_eval/strategies/express.py, eval/a2ui_eval/strategies/elemental.py, eval/a2ui_eval/strategies/direct.py).
    • Implemented generic.py containing a unified format solver (format_system_prompt and compile_format_payload) that polymorphically initializes and executes the target format strategy.
  • Payload Extraction and Parsing:
    • Implemented format-aware surface extraction using regex patterns in the generic evaluation solver.
    • Enabled the programmatic experiments API configuration in the evaluation harness to validate A2UI v1.0 specifications.
  • Scorers & Infrastructure:
    • Updated scorers to use the new a2ui.strategies.schema module.
    • Refactored dataset.py and tasks.py to support the parameterized solvers.
  • Testing:
    • Expanded test_strategies.py to verify prompt generation and response parsing across all strategies.

Impact & Risks

  • None. This only affects the evaluation pipeline and tests.

Testing

Verify the changes by running both SDK and evaluation unit tests:

uv run pytest agent_sdks/python/a2ui_agent/tests/
uv run pytest eval/tests/

gemini-code-assist[bot]

This comment was marked as resolved.

@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 41fbfab to 290d6bf Compare July 10, 2026 02:10
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 290261f to 4dfd8ee Compare July 10, 2026 02:10
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 290d6bf to c8d0884 Compare July 10, 2026 02:21
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 4dfd8ee to 858737e Compare July 10, 2026 02:27
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from c8d0884 to 8245555 Compare July 10, 2026 02:32
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 858737e to eb86792 Compare July 10, 2026 02:34
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 8245555 to f90558e Compare July 10, 2026 02:36
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch 2 times, most recently from eefb752 to 0da428a Compare July 10, 2026 02:37
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 34a0c18 to 0101648 Compare July 10, 2026 02:38
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 0da428a to c5aedc7 Compare July 10, 2026 02:38
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from c5aedc7 to 9aab985 Compare July 10, 2026 03:25
Comment thread eval/a2ui_eval/strategies/generic.py Outdated
default_surface_id = allowed_surface_ids[0] if allowed_surface_ids else "main"

surface_id = default_surface_id
extracted_ids = fmt.extract_surface_ids(completion)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand why we need this dance. Either the surfaceId is dynamically set by the LLM, in which case we read it from the inference format as part of the parsing process, or else its injected from outside, in which case we supply it to the parser. Are you sure we need all this dynamic logic?

@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 9dffbf9 to e4bd324 Compare July 10, 2026 16:48
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 9aab985 to 2ebd962 Compare July 10, 2026 16:49
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from e4bd324 to 6238957 Compare July 10, 2026 16:51
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 2ebd962 to 4630c1c Compare July 10, 2026 16:51
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 6238957 to 82bd501 Compare July 10, 2026 16:56
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 4630c1c to 4246093 Compare July 10, 2026 16:57
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 82bd501 to 22aee2f Compare July 10, 2026 19:21
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 4246093 to e950852 Compare July 10, 2026 19:22
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 22aee2f to 722d6d8 Compare July 10, 2026 19:27
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from e950852 to 0a963d3 Compare July 10, 2026 19:28
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 722d6d8 to dad2209 Compare July 13, 2026 18:02
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 0a963d3 to dc5aa46 Compare July 13, 2026 18:02
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from dad2209 to 33fa342 Compare July 13, 2026 19:45
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from dc5aa46 to 0f931e4 Compare July 13, 2026 19:45
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 33fa342 to ee16f4e Compare July 13, 2026 19:53
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 0f931e4 to e28ad91 Compare July 13, 2026 19:55
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from ee16f4e to 0b0c8ce Compare July 13, 2026 19:58
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from e28ad91 to b090b6e Compare July 13, 2026 19:58
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 0b0c8ce to 4eb6dc8 Compare July 13, 2026 20:08
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from b090b6e to 53f1607 Compare July 13, 2026 20:08
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 4eb6dc8 to 1010526 Compare July 13, 2026 20:19
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch from 53f1607 to 2b4563a Compare July 13, 2026 20:20
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-strategies branch from 1010526 to b91234b Compare July 13, 2026 20:38
@gspencergoog
gspencergoog force-pushed the unified-inference-formats-eval branch 2 times, most recently from a1bee73 to 77a52dd Compare July 13, 2026 21:08
@gspencergoog
gspencergoog merged commit 77a52dd into unified-inference-formats-strategies Jul 13, 2026
11 of 12 checks passed
@gspencergoog
gspencergoog deleted the unified-inference-formats-eval branch July 13, 2026 21:12
@github-project-automation github-project-automation Bot moved this from Todo to Done in A2UI Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants