Skip to content

Commit 18b312b

Browse files
committed
sessions: add backend replay consistency test harness
Add replay consistency coverage for session events, state updates, memory operations, summary compression, and failure recovery across InMemory, SQLite, Mock Redis, and optional real Redis backends. Fixes #89 RELEASE NOTES: NONE
1 parent f2a34ff commit 18b312b

13 files changed

Lines changed: 1110 additions & 0 deletions

docs/replay_consistency_design.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Session / Memory / Summary 回放一致性设计
2+
3+
## 归一化策略
4+
5+
测试会先把各后端读回的数据转换为统一快照。时间戳和自动生成的 ID 不参与比较;字典按键排序,集合和 memory 结果转为稳定顺序。事件顺序、state、记忆内容及工具调用参数仍进行严格比较。
6+
7+
## Summary 比较策略
8+
9+
摘要文本统一大小写、空白和中英文标点后再比较,仅消除不影响语义的展示差异。摘要所属 session、版本、更新时间、覆盖事件范围以及截断后保留的事件仍严格比较,用于发现摘要丢失、错误覆盖或跨 session 绑定。
10+
11+
## 允许差异
12+
13+
确实由存储机制产生的差异,必须在样例的 `allowed_diff` 中按完整字段路径声明。允许差异仍会写入报告并明确标记,不会通过忽略整个对象或某类字段来掩盖其他异常。
14+
15+
## 后端接入方式
16+
17+
InMemory 作为比较基准,默认接入内存 SQLite 和 Mock Redis,无需额外服务即可运行。设置 `REPLAY_REDIS_URL` 后追加真实 Redis,设置 `REPLAY_SQL_URL` 可替换 SQL 地址;设置 `REPLAY_LIGHTWEIGHT=1` 时仅执行 InMemory 回放,并要求三十秒内完成。
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"01_single_turn","operations":[{"op":"append","author":"user","text":"hello"},{"op":"append","author":"agent","text":"hello back"}]}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{"id":"02_multi_turn","operations":[{"op":"append","author":"user","text":"first question"},
2+
{"op":"append","author":"agent","text":"first answer"},
3+
{"op":"append","author":"user","text":"second question"},
4+
{"op":"append","author":"agent","text":"second answer"}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"03_tool_call","operations":[{"op":"append","author":"agent","type":"function_call","name":"weather","args":{"city":"Shenzhen"}},{"op":"append","author":"tool","type":"function_response","name":"weather","response":{"temperature":28}}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"04_session_state","operations":[{"op":"append","author":"agent","text":"set mode","state_delta":{"mode":"draft"}},{"op":"append","author":"agent","text":"overwrite mode","state_delta":{"mode":"final"}},{"op":"append","author":"agent","text":"set count","state_delta":{"count":2}}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"05_scoped_state","operations":[{"op":"append","author":"agent","text":"set scoped state","state_delta":{"session_key":"session","app:theme":"dark","user:locale":"zh-CN","temp:request_id":"temporary"}}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"06_memory","operations":[{"op":"append","author":"user","text":"I prefer tea in the morning"},{"op":"append","author":"agent","text":"I will remember your morning drink preference"},{"op":"append","author":"user","text":"I dislike very sweet drinks"},{"op":"append","author":"agent","text":"I will avoid sweet drink recommendations"},{"op":"append","author":"user","text":"Recommend something for tomorrow"},{"op":"append","author":"agent","text":"I will recommend unsweetened tea"},{"op":"summary","text":"User prefers unsweetened tea in the morning and dislikes very sweet drinks.","keep_recent":2},{"op":"store_memory"},{"op":"search_memory","query":"tea"}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"07_summary_create_update","expected_status":"different","operations":[{"op":"append","author":"user","text":"I need help planning a Shenzhen trip"},{"op":"append","author":"agent","text":"What dates and budget do you have"},{"op":"append","author":"user","text":"I will travel next weekend with a small budget"},{"op":"append","author":"agent","text":"Train travel may be suitable"},{"op":"append","author":"user","text":"I prefer a hotel near the station"},{"op":"append","author":"agent","text":"I will remember the station hotel preference"},{"op":"append","author":"user","text":"Please also check weather"},{"op":"append","author":"agent","type":"function_call","name":"weather","args":{"city":"Shenzhen"}},{"op":"append","author":"tool","type":"function_response","name":"weather","response":{"temperature":28,"condition":"sunny"}},{"op":"summary","text":"User is planning a budget Shenzhen trip next weekend, prefers train travel and a station hotel, and requested weather information.","keep_recent":3},{"op":"append","author":"agent","text":"Shenzhen will be sunny and warm"},{"op":"append","author":"user","text":"Give me a final short itinerary"},{"op":"append","author":"agent","text":"Take the train, stay near the station, and pack light clothes"},{"op":"summary","text":"Budget Shenzhen trip: train travel, station hotel, sunny 28 degree weather, and a short itinerary were discussed.","keep_recent":3}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"08_summary_truncation","expected_status":"different","operations":[{"op":"append","author":"user","text":"I am learning Python for data analysis"},{"op":"append","author":"agent","text":"Start with Python basics and pandas"},{"op":"append","author":"user","text":"I can study four hours each week"},{"op":"append","author":"agent","text":"Use a small weekly practice schedule"},{"op":"append","author":"user","text":"I prefer practical exercises"},{"op":"append","author":"agent","text":"Practice with CSV analysis projects"},{"op":"append","author":"user","text":"What should I study next"},{"op":"append","author":"agent","text":"Next learn plotting with matplotlib"},{"op":"summary","text":"User is learning Python data analysis, has four hours weekly, and prefers practical pandas and CSV exercises.","keep_recent":3,"store_historical":true},{"op":"append","author":"user","text":"Please add one visualization exercise"},{"op":"append","author":"agent","text":"Create a matplotlib chart from a CSV file"}]}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"id":"09_duplicate_store","operations":[{"op":"append","author":"user","text":"preferred color blue"},{"op":"store_memory"},{"op":"store_memory"},{"op":"search_memory","query":"blue"}]}

0 commit comments

Comments
 (0)