Referring to
|
def inject_context() -> Solver: |
|
"""Solver that injects evaluation context into the user prompt.""" |
|
async def solve(state: TaskState, generate: Generate) -> TaskState: |
|
context = state.metadata.get('context') |
|
if context: |
|
for msg in state.messages: |
|
if msg.role == "user": |
|
msg.content = f"Context:\n{context}\n\n{msg.content}" |
|
break |
|
return state |
|
|
|
return solve |
This seems to be only used for one eval - See contextAwareUI in the decrypted eval/datasets/v0_9_prompts.yaml.
I think we can get away with dropping this custom solver, and just inlining the context string in the promptText?
cc @gspencergoog
Referring to
A2UI/eval/a2ui_eval/solvers.py
Lines 95 to 106 in ade478f
This seems to be only used for one eval - See
contextAwareUIin the decrypted eval/datasets/v0_9_prompts.yaml.I think we can get away with dropping this custom solver, and just inlining the
contextstring in thepromptText?cc @gspencergoog