Skip to content

test: add Session/Memory/Summary replay consistency harness#216

Open
Thoth-hyx wants to merge 2 commits into
trpc-group:mainfrom
Thoth-hyx:feat
Open

test: add Session/Memory/Summary replay consistency harness#216
Thoth-hyx wants to merge 2 commits into
trpc-group:mainfrom
Thoth-hyx:feat

Conversation

@Thoth-hyx

Copy link
Copy Markdown

Summary

This PR adds a reusable cross-backend replay consistency harness for Session, Memory, and Session Summary behavior.

The harness executes the same standardized operation traces against InMemory and SQLite-backed services, normalizes non-business fields, compares the replay results, and generates a field-level JSON diff report.

Closes #

Changes

  • Add 11 replay cases covering:
    • Single-turn and multi-turn conversations
    • Function calls and responses
    • Session, user, app, and temporary state updates
    • Memory storage and retrieval
    • Multi-session memory
    • Summary generation and update
    • Summary persistence and event truncation
    • Summary ownership isolation
    • Failed/retried and duplicate writes
  • Add deterministic mock-based summary generation.
  • Normalize generated IDs, timestamps, empty parts, collection representations, and dictionary ordering.
  • Compare summary text separately from metadata such as version, update time, compression counts, and session ownership.
  • Require backend-scoped and path-specific allowed_diff entries with explanations.
  • Generate session_memory_summary_diff_report.json with:
    • Session ID
    • Event index or summary ID
    • Field path
    • Values from both backends
  • Support:
    • REPLAY_MODE=inmemory for InMemory-only runs
    • Default InMemory + SQLite lightweight mode
    • Environment-controlled integration mode
  • Add 10 mutation tests for intentionally injected inconsistencies.

No production code under trpc_agent_sdk/sessions is changed.

Validation

  • Replay consistency tests: 24/24 passed
  • Normal replay cases: 11/11 passed
  • Injected inconsistency detection: 10/10
  • Summary loss, overwrite, and wrong-session detection: 100%
  • Normal-case false-positive rate: 0%
  • Lightweight replay runtime: approximately 0.26 seconds
  • Full tests/sessions regression suite: 361/361 passed
  • Flake8: passed

How to run

# Default lightweight mode: InMemory + SQLite
pytest tests/sessions/test_replay_consistency.py -q

# InMemory-only mode
REPLAY_MODE=inmemory pytest tests/sessions/test_replay_consistency.py -q

# Integration mode
REPLAY_MODE=integration \
REPLAY_SQL_URL=<sql-url> \
REPLAY_REDIS_URL=<redis-url> \
pytest tests/sessions/test_replay_consistency.py -q

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@8080800). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #216   +/-   ##
==========================================
  Coverage        ?   87.92015%           
==========================================
  Files           ?         479           
  Lines           ?       44984           
  Branches        ?           0           
==========================================
  Hits            ?       39550           
  Misses          ?        5434           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@Thoth-hyx

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Rook1ex added a commit to trpc-group/cla-database that referenced this pull request Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant