Skip to content

Commit 5a369a2

Browse files
authored
Fix retrieval prompty to prevent nesting of reason/score/status under properties (#5055)
Inline the properties object and add explicit guidance so the LLM emits reason, score, and status as top-level keys instead of nesting them inside properties.
1 parent 8ea3e98 commit 5a369a2

1 file changed

Lines changed: 4 additions & 5 deletions

File tree

assets/evaluators/builtin/retrieval/evaluator/retrieval.prompty

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ CONTEXT: {{context}}
8686
## Please provide your assessment Score for the previous CONTEXT in relation to the QUERY based on the Definitions above.
8787

8888
Output a JSON object with:
89-
1) properties: an object containing a "thought_chain" string. To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your thought_chain with "Let's think step by step:".
89+
1) properties: an object containing ONLY a "thought_chain" string (do not include any other keys here). To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your thought_chain with "Let's think step by step:".
9090
2) reason: a very short explanation of why you think the input Data should get that Score.
9191
3) score: based on your previous analysis, provide your Score. The Score you give MUST be an integer score (i.e., "1", "2"...) based on the levels of the definitions.
9292
4) status: "completed" if evaluation was performed, or "skipped" if evaluation could not be performed.
@@ -98,12 +98,11 @@ If the QUERY or CONTEXT is empty or not provided, return status "skipped" immedi
9898
```
9999

100100
The reason should always precede the score and should clearly justify the score based on the rubric definitions.
101-
Response format exactly as follows:
101+
102+
Response format exactly as follows ("reason", "score", and "status" MUST be top-level keys, NOT nested inside "properties". Note that "properties" is a single-line inline object — close its `}` immediately after `thought_chain` and then continue with the remaining top-level keys). Stop generating immediately after the final closing brace:
102103

103104
{
104-
"properties": {
105-
"thought_chain": "Let's think step by step: <brief step-by-step analysis>"
106-
},
105+
"properties": { "thought_chain": "Let's think step by step: <brief step-by-step analysis>" },
107106
"reason": "<15-60 words>",
108107
"score": <1-5>,
109108
"status": "completed"

0 commit comments

Comments
 (0)