Skip to content

Commit d7d41e7

Browse files
authored
test: stabilize two LLM-flake tests surfaced after PR #1469 (#1774)
* test: stabilize two LLM-flake tests surfaced after PR #1469 1. test_high_skepticism_response_is_more_hedged_than_low (hs_llm_core): The source claim was "Sam is *supposedly* the most productive engineer ...". The built-in hedge ("supposedly") primes both low- and high-skepticism reflects to echo it, shrinking the gap the judge has to detect. Rephrasing the claim as a direct assertion gives the disposition room to matter — high-skepticism should now hedge while low-skepticism states it directly. 2. test_comprehensive_multi_dimension (was hs_llm_mat): Module-level marker is hs_llm_core; this method was overriding to hs_llm_mat, which sent it through the bedrock/nova-2-lite weak model. That model consistently drops one of the two required dimensions (emotional or preferential) and fails the judge. This is a quality assertion, not a provider-compatibility check, so it belongs in the single-strong-provider tier (matching the pattern PR #1469 used). * test: give skepticism test something to actually be skeptical of CI on the first fix attempt still failed identically — both low- and high-skepticism reflects produced "Sam is considered the most productive engineer..." on gemini-2.5-flash-lite. Root cause: with a single assertive claim and no contradicting signal, skepticism has nothing to express. The disposition trait can only show up when there's tension between facts to weigh differently. Add one piece of contradicting evidence ("Sam's manager noted Sam had missed two deadlines last quarter."). Now skepticism=5 should acknowledge the tension while skepticism=1 should defer to the headline claim. Updated the judge criteria and context accordingly.
1 parent 262d489 commit d7d41e7

2 files changed

Lines changed: 31 additions & 10 deletions

File tree

hindsight-api-slim/tests/test_fact_extraction_quality.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -365,10 +365,17 @@ async def test_evaluative_preferential_dimension(self):
365365
msg=f"Evaluative/preferential dimension should be preserved. Facts: {[f.fact for f in facts]}",
366366
)
367367

368-
@pytest.mark.hs_llm_mat
369368
@pytest.mark.asyncio
370369
async def test_comprehensive_multi_dimension(self):
371-
"""Test a realistic scenario with multiple dimensions in one fact."""
370+
"""Test a realistic scenario with multiple dimensions in one fact.
371+
372+
Inherits the module-level `hs_llm_core` marker rather than running on
373+
the full `hs_llm_mat` matrix — the weakest matrix providers (notably
374+
bedrock/nova-2-lite) drop one of the two required dimensions (emotional
375+
or preferential) and fail the judge. This is a quality assertion, not
376+
a provider-compatibility check, so single-strong-provider is the right
377+
tier.
378+
"""
372379
text = """
373380
I was thrilled to receive such positive feedback on my presentation yesterday!
374381
I wasn't sure if my approach would resonate, but the audience seemed enthusiastic.

hindsight-api-slim/tests/test_quality_integration.py

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,15 @@ async def test_high_skepticism_response_is_more_hedged_than_low(self, memory: Me
160160
This catches the case where the disposition trait isn't wired into the prompt
161161
at all — both responses would then look equally confident.
162162
"""
163-
claim = "Sam is supposedly the most productive engineer on the team by a wide margin."
163+
# Skepticism only has something to express when there's *something to
164+
# be skeptical of*. With a single assertive claim and no contradicting
165+
# signal, both low- and high-skepticism reflects converge on the same
166+
# restatement (verified on gemini-2.5-flash-lite). We store one
167+
# assertive claim plus one piece of contradicting evidence so the
168+
# high-skepticism bank can flag the tension while the low-skepticism
169+
# bank still defers to the headline claim.
170+
claim = "Sam is the most productive engineer on the team by a wide margin."
171+
contradicting = "Sam's manager noted Sam had missed two deadlines last quarter."
164172
query = "What can you tell me about Sam's productivity?"
165173

166174
bank_low = f"test-disposition-low-{uuid.uuid4().hex[:8]}"
@@ -174,6 +182,7 @@ async def test_high_skepticism_response_is_more_hedged_than_low(self, memory: Me
174182
request_context=request_context,
175183
)
176184
await memory.retain_async(bank_id=bank_id, content=claim, request_context=request_context)
185+
await memory.retain_async(bank_id=bank_id, content=contradicting, request_context=request_context)
177186

178187
low_result = await memory.reflect_async(bank_id=bank_low, query=query, request_context=request_context)
179188
high_result = await memory.reflect_async(bank_id=bank_high, query=query, request_context=request_context)
@@ -193,15 +202,20 @@ async def test_high_skepticism_response_is_more_hedged_than_low(self, memory: Me
193202
response=comparison,
194203
criteria=(
195204
"Response A shows more skepticism than Response B about the productivity "
196-
"claim — A uses more hedging language ('apparently', 'reportedly', 'might', "
197-
"'allegedly', etc.) or more explicitly flags the claim as unverified, while "
198-
"B states it more directly or with less qualification. If the two responses "
199-
"show the same level of skepticism, the criterion is NOT met."
205+
"claim — A acknowledges the tension between 'most productive' and the missed "
206+
"deadlines, uses hedging language ('apparently', 'mixed signals', 'might', etc.), "
207+
"or more explicitly flags the headline claim as unverified or qualified. "
208+
"B states the 'most productive' claim more directly, downplays the missed "
209+
"deadlines, or treats the headline claim as authoritative. If both responses "
210+
"show the same level of skepticism — e.g. both restate the claim with similar "
211+
"qualification — the criterion is NOT met."
200212
),
201213
context=(
202-
"Both banks stored the same claim ('Sam is supposedly the most productive "
203-
"engineer...') and were asked the same query. The only difference is "
204-
"their skepticism disposition trait. A: skepticism=5, B: skepticism=1."
214+
"Both banks stored TWO facts: the headline claim 'Sam is the most productive "
215+
"engineer on the team by a wide margin.' AND a contradicting signal 'Sam's "
216+
"manager noted Sam had missed two deadlines last quarter.' They were asked "
217+
"the same query. The only configuration difference is the skepticism "
218+
"disposition trait. A: skepticism=5, B: skepticism=1."
205219
),
206220
msg=(
207221
f"Disposition should make response A more skeptical than B.\n"

0 commit comments

Comments
 (0)