refactor(eval): unify evaluation strategy solvers using generic InferenceFormat solver#1983
Merged
gspencergoog merged 0 commit intoJul 13, 2026
Conversation
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 02:10
41fbfab to
290d6bf
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 02:10
290261f to
4dfd8ee
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 02:21
290d6bf to
c8d0884
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 02:27
4dfd8ee to
858737e
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 02:32
c8d0884 to
8245555
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 02:34
858737e to
eb86792
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 02:36
8245555 to
f90558e
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
2 times, most recently
from
July 10, 2026 02:37
eefb752 to
0da428a
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 02:38
34a0c18 to
0101648
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 02:38
0da428a to
c5aedc7
Compare
This was referenced Jul 10, 2026
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 03:25
c5aedc7 to
9aab985
Compare
| default_surface_id = allowed_surface_ids[0] if allowed_surface_ids else "main" | ||
|
|
||
| surface_id = default_surface_id | ||
| extracted_ids = fmt.extract_surface_ids(completion) |
Collaborator
There was a problem hiding this comment.
I don't quite understand why we need this dance. Either the surfaceId is dynamically set by the LLM, in which case we read it from the inference format as part of the parsing process, or else its injected from outside, in which case we supply it to the parser. Are you sure we need all this dynamic logic?
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 16:48
9dffbf9 to
e4bd324
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 16:49
9aab985 to
2ebd962
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 16:51
e4bd324 to
6238957
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 16:51
2ebd962 to
4630c1c
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 16:56
6238957 to
82bd501
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 16:57
4630c1c to
4246093
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 19:21
82bd501 to
22aee2f
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 19:22
4246093 to
e950852
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 10, 2026 19:27
22aee2f to
722d6d8
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 10, 2026 19:28
e950852 to
0a963d3
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 18:02
722d6d8 to
dad2209
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 13, 2026 18:02
0a963d3 to
dc5aa46
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 19:45
dad2209 to
33fa342
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 13, 2026 19:45
dc5aa46 to
0f931e4
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 19:53
33fa342 to
ee16f4e
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 13, 2026 19:55
0f931e4 to
e28ad91
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 19:58
ee16f4e to
0b0c8ce
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 13, 2026 19:58
e28ad91 to
b090b6e
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 20:08
0b0c8ce to
4eb6dc8
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 13, 2026 20:08
b090b6e to
53f1607
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 20:19
4eb6dc8 to
1010526
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
from
July 13, 2026 20:20
53f1607 to
2b4563a
Compare
gspencergoog
force-pushed
the
unified-inference-formats-strategies
branch
from
July 13, 2026 20:38
1010526 to
b91234b
Compare
gspencergoog
force-pushed
the
unified-inference-formats-eval
branch
2 times, most recently
from
July 13, 2026 21:08
a1bee73 to
77a52dd
Compare
gspencergoog
merged commit Jul 13, 2026
77a52dd
into
unified-inference-formats-strategies
11 of 12 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates the unified
InferenceStrategyandParserabstractions into the A2UI evaluation framework, replacing redundant solver scripts with a single generic format solver.Changes
eval/a2ui_eval/strategies/express.py,eval/a2ui_eval/strategies/elemental.py,eval/a2ui_eval/strategies/direct.py).generic.pycontaining a unified format solver (format_system_promptandcompile_format_payload) that polymorphically initializes and executes the target format strategy.experimentsAPI configuration in the evaluation harness to validate A2UI v1.0 specifications.a2ui.strategies.schemamodule.dataset.pyandtasks.pyto support the parameterized solvers.test_strategies.pyto verify prompt generation and response parsing across all strategies.Impact & Risks
Testing
Verify the changes by running both SDK and evaluation unit tests: