Skip to content

Commit 2fc7756

Browse files
DNKYrDNKYr
committed
docs: add replay consistency harness design notes
Add DESIGN.md (English) and DESIGN.zh_CN.md (Chinese) for the Session/Memory/Summary multi-backend replay consistency testing framework. Covers normalization strategy, summary comparison approach, allowed difference rules, backend integration, and harness self-testing strategy. Updates #89 RELEASE NOTES: NONE
1 parent 4336564 commit 2fc7756

2 files changed

Lines changed: 244 additions & 0 deletions

File tree

‎docs/replay_harness/DESIGN.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# Session / Memory / Summary Replay Consistency Harness — Design
2+
3+
## Objective
4+
5+
Validate that InMemory, SQL, and Redis backends for Session, Memory, and Summary
6+
services produce equivalent results when driven by identical operation sequences.
7+
This harness replays standardized input trajectories through multiple backends
8+
and generates a structured diff report.
9+
10+
## Architecture
11+
12+
```
13+
replay_cases/*.json → ReplayEngine → BackendResult
14+
│
15+
┌─────────┴─────────┐
16+
Backend A Backend B
17+
(InMemory) (SQL / Redis)
18+
│ │
19+
BackendResult BackendResult
20+
│ │
21+
└────────┬───────────┘
22+
▼
23+
_normalizer.py
24+
(strip timestamps, IDs, sort keys)
25+
│
26+
NormalizedResult A + B
27+
│
28+
▼
29+
_comparator.py
30+
(pairwise diff events, state, memory, summary)
31+
│
32+
▼
33+
DiffReport (JSON)
34+
```
35+
36+
Each replay case is a JSON file describing a sequence of operations:
37+
`create_session`, `append_event`, `update_state`, `inject_summary`,
38+
`store_memory`, `search_memory`, `read_back`.
39+
40+
The ReplayEngine executes the same sequence against two backends in parallel,
41+
collecting raw results. These are then normalized and compared.
42+
43+
## Normalization Strategy
44+
45+
Fields that differ across backends by design are normalized before comparison:
46+
47+
| Field | Strategy |
48+
|------------------------|----------------------------------------------------------|
49+
| `event.id` | Stripped (auto-generated UUID per backend) |
50+
| `event.timestamp` | Replaced with sequential index (0, 1, 2, ...) |
51+
| `session.last_update_time` | Replaced with sentinel `0.0` |
52+
| `summary_timestamp` | Replaced with sentinel `0.0` |
53+
| `memory_entry.timestamp` | Stripped |
54+
| dict key ordering | Re-serialized with `sort_keys=True` |
55+
| `invocation_id` | Stripped (invocation-scoped, not backend-scoped) |
56+
| `branch`, `request_id` | Stripped (runtime metadata, not persisted uniformly) |
57+
58+
## Summary Comparison Strategy
59+
60+
The harness distinguishes two levels of summary correctness:
61+
62+
1. **Content semantic consistency** — the `summary_text` is compared after
63+
whitespace normalization. Minor formatting differences that do not change
64+
meaning are treated as allowed diffs on a per-backend-pair basis.
65+
66+
2. **Metadata integrity** — `session_id`, `original_event_count`, and
67+
`compressed_event_count` must match exactly across backends. Any deviation
68+
in these fields is an unconditional failure. Three classes of summary bug
69+
are explicitly detected:
70+
- **Summary loss** — summary present in backend A, absent in backend B.
71+
- **Summary overwrite** — summary exists but with wrong `session_id`.
72+
- **Wrong session affiliation** — summary is stored under the wrong session
73+
key in the underlying cache or storage.
74+
75+
Summary injection into the `SummarizerSessionManager._summarizer_cache`
76+
bypasses the LLM, since the harness tests storage consistency, not
77+
summarization model quality.
78+
79+
## Allowed Differences
80+
81+
Not all field-level differences indicate a bug. Known, documented divergences
82+
are captured in `AllowedDiff` rules:
83+
84+
```
85+
allowed_diffs = {
86+
"inmem_vs_sql": [
87+
{"field": "events[*].function_calls[*].args", "reason": "SQL serializes
88+
JSON args differently from InMemory dict round-trip"},
89+
],
90+
"inmem_vs_redis": [
91+
{"field": "events[*].timestamp_precision",
92+
"reason": "Redis stores timestamps as float strings with limited precision"},
93+
],
94+
}
95+
```
96+
97+
Each rule includes the backend pair, the field path, and a justification.
98+
Diffs matching an allowed rule are suppressed from the failure count but
99+
still appear in the report tagged `allowed: true`.
100+
101+
## Backend Integration
102+
103+
| Backend | Availability | Activation |
104+
|----------|-----------------|---------------------------------------------|
105+
| InMemory | Always | Direct instantiation |
106+
| SQL | Always (sqlite) | `sqlite:///:memory:` — no external deps |
107+
| Redis | Opt-in | `TRPC_REDIS_URL` env var; skipped otherwise |
108+
109+
The lightweight mode (InMemory + SQL) runs in CI without external services.
110+
Redis integration mode requires the environment variable to be set; when
111+
absent, Redis test pairs are skipped with a descriptive message.
112+
113+
## Diff Report
114+
115+
The report (`session_memory_summary_diff_report.json`) contains:
116+
117+
- **run metadata**: run ID, timestamp, backends tested.
118+
- **per-case results**: status (pass/fail/error), list of `DiffEntry` objects.
119+
- **summary**: total/pass/fail counts and false-positive rate.
120+
121+
Each `DiffEntry` pinpoints: backend pair, session ID, event index (or summary
122+
ID), category (events/state/memory/summary), full dotted field path, and the
123+
two conflicting values.
124+
125+
## Testing the Harness Itself
126+
127+
- `test_replay_normalizer.py` — unit tests for every normalization function.
128+
- `test_replay_comparator.py` — unit tests for diff logic, including targeted
129+
tests for summary loss, overwrite, and mis-affiliation detection.
130+
- `test_replay_report.py` — unit tests for report generation and serialization.
131+
- `test_replay_consistency.py` — E2E tests running all 10 replay cases through
132+
backend pairs, asserting normal cases produce ≤5% false positives and anomaly
133+
cases are 100% detected.

‎docs/replay_harness/DESIGN.zh_CN.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Session / Memory / Summary 回放一致性测试框架 — 设计说明
2+
3+
## 目标
4+
5+
验证 InMemory、SQL、Redis 三种后端的 Session、Memory 和 Summary 服务在相同操作序列下是否产出一致的结果。该框架使用标准化输入轨迹驱动多个后端，并生成结构化差异报告。
6+
7+
## 架构
8+
9+
```
10+
replay_cases/*.json → ReplayEngine → BackendResult
11+
│
12+
┌─────────┴─────────┐
13+
后端 A 后端 B
14+
(InMemory) (SQL / Redis)
15+
│ │
16+
BackendResult BackendResult
17+
│ │
18+
└────────┬───────────┘
19+
▼
20+
_normalizer.py
21+
（去除时间戳、ID，排序键）
22+
│
23+
NormalizedResult A + B
24+
│
25+
▼
26+
_comparator.py
27+
（逐对比较事件、状态、记忆、摘要）
28+
│
29+
▼
30+
DiffReport（JSON 差异报告）
31+
```
32+
33+
每个回放用例是一个 JSON 文件，描述一系列操作：
34+
`create_session`、`append_event`、`update_state`、`inject_summary`、
35+
`store_memory`、`search_memory`、`read_back`。
36+
37+
ReplayEngine 将相同的操作序列在两个后端上并行执行，收集原始结果后进行归一化和比较。
38+
39+
## 归一化策略
40+
41+
对后端间必然不同的字段，在比较前进行归一化：
42+
43+
| 字段 | 策略 |
44+
|---------------------------|-----------------------------------------------|
45+
| `event.id` | 去除（各后端自动生成 UUID） |
46+
| `event.timestamp` | 替换为顺序索引（0, 1, 2, ...） |
47+
| `session.last_update_time` | 替换为哨兵值 `0.0` |
48+
| `summary_timestamp` | 替换为哨兵值 `0.0` |
49+
| `memory_entry.timestamp` | 去除 |
50+
| dict 键顺序 | 使用 `sort_keys=True` 重新序列化 |
51+
| `invocation_id` | 去除（调用范围标识，非后端相关） |
52+
| `branch`、`request_id` | 去除（运行时元数据，非统一持久化字段） |
53+
54+
## 摘要比较策略
55+
56+
框架区分两层摘要正确性：
57+
58+
1. **内容语义一致性** — `summary_text` 在空白符归一化后进行比较。不影响含义的微小格式差异按后端对允许规则处理。
59+
60+
2. **元数据完整性** — `session_id`、`original_event_count`、`compressed_event_count` 必须在各后端间完全匹配。任何偏差均视为无条件失败。框架明确检测三类摘要缺陷：
61+
- **摘要丢失** — 摘要存在于后端 A，但后端 B 中缺失。
62+
- **摘要覆盖错误** — 摘要存在但 `session_id` 不正确。
63+
- **摘要归属错误** — 摘要在底层缓存或存储中被错误地关联到不正确的 session 键。
64+
65+
摘要通过直接注入 `SummarizerSessionManager._summarizer_cache` 的方式生成，绕过 LLM，因为框架测试的是存储一致性而非摘要模型质量。
66+
67+
## 允许差异
68+
69+
并非所有字段级差异都表示缺陷。已知且有文档记录的差异通过 `AllowedDiff` 规则管理：
70+
71+
```
72+
allowed_diffs = {
73+
"inmem_vs_sql": [
74+
{"field": "events[*].function_calls[*].args",
75+
"reason": "SQL 对 JSON 参数的序列化与 InMemory dict 往返不同"},
76+
],
77+
"inmem_vs_redis": [
78+
{"field": "events[*].timestamp_precision",
79+
"reason": "Redis 将时间戳存储为精度有限的浮点字符串"},
80+
],
81+
}
82+
```
83+
84+
每条规则包含后端对、字段路径和原因说明。匹配允许规则的差异不计入失败统计，但仍会在报告中以 `allowed: true` 标记呈现。
85+
86+
## 后端接入方式
87+
88+
| 后端 | 可用性 | 激活方式 |
89+
|----------|--------------------|----------------------------------------------|
90+
| InMemory | 始终可用 | 直接实例化 |
91+
| SQL | 始终可用（sqlite） | `sqlite:///:memory:` — 无外部依赖 |
92+
| Redis | 按需启用 | 设置 `TRPC_REDIS_URL` 环境变量；否则跳过 |
93+
94+
轻量模式（InMemory + SQL）无需外部服务即可在 CI 中运行。Redis 集成模式需要设置相应的环境变量，未设置时跳过 Redis 测试对并给出明确提示。
95+
96+
## 差异报告
97+
98+
报告（`session_memory_summary_diff_report.json`）包含：
99+
100+
- **运行元数据**：运行 ID、时间戳、已测试的后端列表。
101+
- **逐用例结果**：状态（pass/fail/error）、`DiffEntry` 列表。
102+
- **汇总**：总计/通过/失败数量和误报率。
103+
104+
每个 `DiffEntry` 精确定位：后端对、session ID、事件索引（或摘要 ID）、类别（events/state/memory/summary）、完整的点分隔字段路径以及两个冲突值。
105+
106+
## 框架自身的测试
107+
108+
- `test_replay_normalizer.py` — 每个归一化函数的单元测试。
109+
- `test_replay_comparator.py` — 差异逻辑的单元测试，包含针对摘要丢失、覆盖错误和归属错误检测的专项测试。
110+
- `test_replay_report.py` — 报告生成和序列化的单元测试。
111+
- `test_replay_consistency.py` — 端到端测试，通过后端对运行全部 10 个回放用例，验证正常用例误报率 ≤ 5%，异常用例 100% 检出。

0 commit comments

Comments
 (0)