Commit d7d41e7
authored
* test: stabilize two LLM-flake tests surfaced after PR #1469
1. test_high_skepticism_response_is_more_hedged_than_low (hs_llm_core):
The source claim was "Sam is *supposedly* the most productive engineer
...". The built-in hedge ("supposedly") primes both low- and
high-skepticism reflects to echo it, shrinking the gap the judge has
to detect. Rephrasing the claim as a direct assertion gives the
disposition room to matter — high-skepticism should now hedge while
low-skepticism states it directly.
2. test_comprehensive_multi_dimension (was hs_llm_mat):
Module-level marker is hs_llm_core; this method was overriding to
hs_llm_mat, which sent it through the bedrock/nova-2-lite weak model.
That model consistently drops one of the two required dimensions
(emotional or preferential) and fails the judge. This is a quality
assertion, not a provider-compatibility check, so it belongs in the
single-strong-provider tier (matching the pattern PR #1469 used).
* test: give skepticism test something to actually be skeptical of
CI on the first fix attempt still failed identically — both low- and
high-skepticism reflects produced "Sam is considered the most productive
engineer..." on gemini-2.5-flash-lite. Root cause: with a single
assertive claim and no contradicting signal, skepticism has nothing to
express. The disposition trait can only show up when there's tension
between facts to weigh differently.
Add one piece of contradicting evidence ("Sam's manager noted Sam had
missed two deadlines last quarter."). Now skepticism=5 should
acknowledge the tension while skepticism=1 should defer to the headline
claim. Updated the judge criteria and context accordingly.
1 parent 262d489 commit d7d41e7
2 files changed
Lines changed: 31 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
365 | 365 | | |
366 | 366 | | |
367 | 367 | | |
368 | | - | |
369 | 368 | | |
370 | 369 | | |
371 | | - | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
372 | 379 | | |
373 | 380 | | |
374 | 381 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
163 | | - | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
164 | 172 | | |
165 | 173 | | |
166 | 174 | | |
| |||
174 | 182 | | |
175 | 183 | | |
176 | 184 | | |
| 185 | + | |
177 | 186 | | |
178 | 187 | | |
179 | 188 | | |
| |||
193 | 202 | | |
194 | 203 | | |
195 | 204 | | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
200 | 212 | | |
201 | 213 | | |
202 | | - | |
203 | | - | |
204 | | - | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
205 | 219 | | |
206 | 220 | | |
207 | 221 | | |
| |||
0 commit comments