Skip to content

fix: reduce flakiness in retrieval config integration test#437

Merged
Hweinstock merged 1 commit into
aws:mainfrom
Hweinstock:fix/flaky-retrieval-config-test
Apr 24, 2026
Merged

fix: reduce flakiness in retrieval config integration test#437
Hweinstock merged 1 commit into
aws:mainfrom
Hweinstock:fix/flaky-retrieval-config-test

Conversation

@Hweinstock

@Hweinstock Hweinstock commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Problem

The integration test test_session_manager_with_retrieval_config_adds_context is flaky (#432). It fails because the relevance score filter (relevance_score=0.7) drops retrieved memories when the semantic similarity between the query "What do I like to eat?" and the extracted preference about sushi scores below 0.7. The test failed 3/3 attempts (original + 2 reruns) in this CI run.

CI logs confirm: Retrieved 1 memories appears but Retrieved X customer context items never does — the relevance filter drops the memory before context injection.

Solution

Two changes to make the test more reliable:

  1. Lower relevance_score from 0.7 to 0.3

  2. Change query from "What do I like to eat?" to "What sushi do I like?" — semantically closer to the stored preference, producing higher and more consistent relevance scores.

Testing

The test validates the same invariant: retrieval config with LTM causes <user_context> to appear in agent messages. The changes only reduce non-determinism in the relevance score path without weakening the assertion.

@codecov-commenter

codecov-commenter commented Apr 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@d8c91cb). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #437   +/-   ##
=======================================
  Coverage        ?   91.06%           
=======================================
  Files           ?       60           
  Lines           ?     5072           
  Branches        ?      775           
=======================================
  Hits            ?     4619           
  Misses          ?      257           
  Partials        ?      196           
Flag Coverage Δ
unittests 91.06% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Hweinstock Hweinstock force-pushed the fix/flaky-retrieval-config-test branch from ff48fe3 to 8f07d61 Compare April 24, 2026 18:58
@Hweinstock Hweinstock marked this pull request as ready for review April 24, 2026 19:17
@Hweinstock Hweinstock requested a review from a team April 24, 2026 19:17

@notgitika notgitika left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks for looking into this!

@Hweinstock Hweinstock merged commit 5dfce96 into aws:main Apr 24, 2026
27 checks passed
@Hweinstock Hweinstock deleted the fix/flaky-retrieval-config-test branch April 24, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants