Skip to content

[evals] inject_context might not be needed #1509

@jiahaog

Description

@jiahaog

Referring to

def inject_context() -> Solver:
"""Solver that injects evaluation context into the user prompt."""
async def solve(state: TaskState, generate: Generate) -> TaskState:
context = state.metadata.get('context')
if context:
for msg in state.messages:
if msg.role == "user":
msg.content = f"Context:\n{context}\n\n{msg.content}"
break
return state
return solve

This seems to be only used for one eval - See contextAwareUI in the decrypted eval/datasets/v0_9_prompts.yaml.

I think we can get away with dropping this custom solver, and just inlining the context string in the promptText?

cc @gspencergoog

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

Status
Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions