test/replaytest: add Session/Memory multi-backend replay consistency framework#2264
test/replaytest: add Session/Memory multi-backend replay consistency framework#2264MUYI-luyu wants to merge 13 commits into
Conversation
Introduce the replaytest package — a reusable harness that drives Session and Memory backends through identical operation sequences and detects cross-backend inconsistencies with field-level diffs. Framework (test/replaytest/): - scenario: ReplayCase types + JSON loader with validation - backends: ReplayBackend, DeterministicSummarizer, factory - driver: RunReplayCase engine supporting all 12 step types - normalizer: CaptureSnapshot with field-level normalization - comparator: CompareSnapshots with recursive path-precise diff - report: WriteDiffReport + HasUnexpectedDiffs Test data (test/testdata/): - case_01_single_turn: single user + assistant event - case_02_multi_turn: six sequential events Tests (test/consistency_test.go): - Load and validate both JSON scenarios - Run InMemory vs SQLite for each case (zero diffs) - Verify snapshot fields in detail Zero LLM dependency — DeterministicSummarizer returns preset text directly, requiring no API keys.
…d events_order_independent - buildEvent: write Timestamp to both event.Event.Timestamp and Response.Timestamp so SummarizeSession→laterBoundary reads non-zero time, preventing SQLite getSummariesList from filtering all summaries - runConcurrentSteps: pre-build events outside mutex so goroutines interleave; lock only GetSession+AppendEvent critical section - normalizeSummaries: omit CutoffAt from replayBoundary (wall-clock) - VerifySpec: add EventsOrderIndependent bool for content-based comparison of non-deterministically ordered events - RunReplayCase: sort normalized events by JSON serialization when EventsOrderIndependent is set
- case_01: single-turn user + assistant text events - case_02: multi-turn (3 rounds, 6 events) - case_03: tool call + tool response with args/extensions - case_04: app/user/session state write/overwrite/delete - case_05: memory add→update→delete with aliases - case_06: summary generation with force update - case_07: summary truncation with boundary verification - case_08: track events with payloads - case_09: concurrent event appends (events_order_independent) - case_10: error recovery (duplicate writes, session re-create)
…report - consistency_test.go: AllCases (10-case InMemory vs SQLite), InjectedInconsistencies (9 types, 100% detection), LightweightMode (≤30s), VerifySpec, round-trip validation - comparator_test.go: recursiveDiff, wildcardMatch, allowedRules, buildDiffContext (event_index, summary_filter_key, track_name) - normalizer_test.go: event ID/timestamp stripping, state normalization, memory sorting, summary boundary, track sorting - report_test.go: WriteDiffReport, DiffReportPath, HasUnexpectedDiffs - README.md: design overview, quick start, lightweight/integration modes, case JSON format, diff report spec, feature coverage table - README_CN.md: Chinese version of the full documentation - sample_diff_report.json: format reference with all context fields - session_memory_summary_track_diff_report.json: full-suite output ([] = zero unexpected diffs across all 10 cases)
These files were left from the initial skeleton commit. Their correct counterparts now live under test/replaytest/ with full coverage.
…iles - session_memory_summary_track_diff_report.json: replace empty [] with 5 illustrative entries covering all sections (events, memory, summary, tracks) and context fields (event_index, memory_id, summary_filter_key, track_name), including one allowed_diff example - Remove test/consistency_test.go and test/testdata/: stale skeleton duplicates now superseded by test/replaytest/ - Remove sample_diff_report.json: merged into the main report above
|
All contributors have signed the CLA ✍️ ✅ |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a replay-consistency harness comparing InMemory and SQLite Session/Memory backends through deterministic scenarios, normalized snapshots, recursive diffs, reports, and ten replay cases. ChangesReplay consistency harness
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
I have read the CLA Document and I hereby sign the CLA |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2264 +/- ##
===================================================
+ Coverage 89.85102% 89.85483% +0.00381%
===================================================
Files 1149 1150 +1
Lines 203321 203417 +96
===================================================
+ Hits 182686 182780 +94
- Misses 12918 12922 +4
+ Partials 7717 7715 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
test/replaytest/consistency_driver.go (1)
26-32: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winGlobal mutable
stepBaseTimeis a flakiness/global-state smell.Re-deriving
stepBaseTimefrom wall-clock time independently on everyRunReplayCasecall (line 52) means the two backend runs for the same case can get different base times if they straddle a one-second boundary — andnormalizeTracks(consistency_normalizer.go) keeps absolute track timestamps rather than stripping them like events do, so this could produce a rare, timing-dependent spurious diff for track-based cases. It's also a package-level global that would silently become a real data race if a future contributor parallelizes case subtests witht.Parallel().Consider computing the reference time once per case (e.g., in the test before invoking both backends) and passing it explicitly into
RunReplayCase/buildEventinstead of a packagevar.全局可变的
stepBaseTime存在不稳定/全局状态问题。
stepBaseTime在每次RunReplayCase调用时都独立从当前时钟重新计算（第52行），意味着同一 case 的两次 backend 运行如果跨越秒边界会得到不同的基准时间——而normalizeTracks保留了绝对时间戳（不像事件那样被剥离），可能导致 track 类用例出现偶发的、与时序相关的虚假差异。这也是一个包级全局变量，若未来有人为 case 子测试添加t.Parallel()，会悄然变成真正的数据竞争。建议每个用例只计算一次参考时间（例如在调用两个 backend 之前），并显式传入
RunReplayCase/buildEvent，而不是使用包级var。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/replaytest/consistency_driver.go` around lines 26 - 32, Remove the package-level mutable stepBaseTime and compute one reference timestamp per test case before running either backend. Pass that timestamp explicitly through RunReplayCase to buildEvent, ensuring both backend runs for the same case reuse the identical value and avoiding shared global state.test/replaytest/consistency_comparator.go (1)
166-188: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftPositional list diffing can cascade misleading diffs on mid-list insert/delete.
diffListcompares elements strictly by index. If backend A and B disagree only on presence of one memory/track/event mid-list (not the last one), every subsequent index will be reported as "missing/extra/mismatched" rather than pinpointing the single real difference — undermining the precise-location goal of this harness. Notably,consistency_test.go's ownmissing_memoryinjection test only removes the last element, which reads as an implicit acknowledgment of this limitation rather than exercising the general case.Consider a content-keyed diff (e.g., match memory entries by
Key/track events by content fingerprint before diffing, similar to hownormalizeMemoriesalready sorts byKey) so a single insert/delete produces a single diff entry.位置式列表比较在中间插入/删除时会产生连锁式误报差异。
diffList严格按索引比较元素。如果两个 backend 只是在列表中间（而非末尾）存在一条记录的差异，后续所有索引都会被报告为"缺失/多余/不一致"，而不是精确定位那一条真正的差异——这与本框架"精确定位差异"的目标相悖。值得注意的是，consistency_test.go中的missing_memory注入测试只删除了最后一个元素，这暗示作者已意识到该局限性，只是回避了一般场景。建议在比较前先按内容做键值对齐（例如按
Key/内容指纹匹配 memory/track 条目，类似normalizeMemories已经按Key排序的做法），使单次插入/删除只产生一条差异记录。🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/replaytest/consistency_comparator.go` around lines 166 - 188, The positional matching in diffList causes cascading diffs when a memory, track, or event is inserted or removed mid-list. Update diffList and its callers to align entries by a stable content key or fingerprint before invoking recursiveDiff, reusing the existing normalizeMemories keying approach where applicable, so unmatched entries produce only their own valueDiff while matched entries are compared regardless of index.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/replaytest/consistency_normalizer.go`:
- Around line 213-236: Update normalizeSummaries when constructing
entry.Boundary from sum.CutoffBoundary() to also copy the boundary’s CutoffAt
value into replayBoundary, preserving the existing Version, FilterKey, and
LastEventID fields.
In `@test/replaytest/consistency_report.go`:
- Around line 19-53: Change defaultReportName and DiffReportPath in
test/replaytest/consistency_report.go (lines 19-53) so unconfigured tests write
to a temporary/build-specific path instead of the tracked fixture name. Update
any default-path assumptions in test/replaytest/consistency_test.go (lines 75-80
and 410-418); test/replaytest/session_memory_summary_track_diff_report.json
(lines 1-81) requires no direct change.
In `@test/replaytest/consistency_scenario.go`:
- Around line 201-205: Extend the Validate method’s step-type validation to also
iterate through each nested step in Concurrent and reject any type absent from
validStepTypes with the same fail-fast error behavior. Preserve the existing
top-level validation and error reporting context.
In `@test/replaytest/consistency_test.go`:
- Around line 142-166: Update TestReplayConsistency_LightweightMode to keep
logging the elapsed replay duration but remove the hard 30-second t.Errorf
failure, so slow or loaded CI runners do not cause nondeterministic test
failures.
---
Nitpick comments:
In `@test/replaytest/consistency_comparator.go`:
- Around line 166-188: The positional matching in diffList causes cascading
diffs when a memory, track, or event is inserted or removed mid-list. Update
diffList and its callers to align entries by a stable content key or fingerprint
before invoking recursiveDiff, reusing the existing normalizeMemories keying
approach where applicable, so unmatched entries produce only their own valueDiff
while matched entries are compared regardless of index.
In `@test/replaytest/consistency_driver.go`:
- Around line 26-32: Remove the package-level mutable stepBaseTime and compute
one reference timestamp per test case before running either backend. Pass that
timestamp explicitly through RunReplayCase to buildEvent, ensuring both backend
runs for the same case reuse the identical value and avoiding shared global
state.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 05c850e9-267d-499f-bbd1-3838818915ed
⛔ Files ignored due to path filters (1)
test/go.sumis excluded by!**/*.sum
📒 Files selected for processing (25)
test/go.modtest/replaytest/README.mdtest/replaytest/README_CN.mdtest/replaytest/comparator_test.gotest/replaytest/consistency_backends.gotest/replaytest/consistency_comparator.gotest/replaytest/consistency_driver.gotest/replaytest/consistency_normalizer.gotest/replaytest/consistency_report.gotest/replaytest/consistency_scenario.gotest/replaytest/consistency_test.gotest/replaytest/normalizer_test.gotest/replaytest/report_test.gotest/replaytest/sample_diff_report.jsontest/replaytest/session_memory_summary_track_diff_report.jsontest/replaytest/testdata/case_01_single_turn.jsontest/replaytest/testdata/case_02_multi_turn.jsontest/replaytest/testdata/case_03_tool_calls.jsontest/replaytest/testdata/case_04_state_updates.jsontest/replaytest/testdata/case_05_memory_crud.jsontest/replaytest/testdata/case_06_summary.jsontest/replaytest/testdata/case_07_summary_truncation.jsontest/replaytest/testdata/case_08_tracks.jsontest/replaytest/testdata/case_09_concurrent.jsontest/replaytest/testdata/case_10_error_recovery.json
Fix golangci-lint failures: the project requires 'any' over
'interface{}' (Go 1.18+) on Go 1.24.4 code.
- Use unique app_name per case to prevent cross-case state leakage - Validate nested step types inside Concurrent blocks recursively - Soften 30s hard assertion to diagnostic warning
- Rename default output to replay_diff_report.json and add to .gitignore - Use sample_diff_report.json as the permanent reference artifact - Remove old session_memory_summary_track_diff_report.json from tracking - Drop init() cleanup hack — the gitignored output no longer dirties the tree
- Change defaultReportName to replay_diff_report.json - Remove init() cleanup hack — no longer needed with gitignored output
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.gitignore:
- Around line 44-45: Update the .gitignore entry for the replay consistency
report to cover both generated locations: the existing
test/replaytest/replay_diff_report.json path and the repository-root
replay_diff_report.json path produced by DiffReportPath().
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 56f56d47-56c2-40f1-aa37-dfca755e4dd0
📒 Files selected for processing (3)
.gitignoretest/replaytest/consistency_report.gotest/replaytest/consistency_test.go
💤 Files with no reviewable changes (1)
- test/replaytest/consistency_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- test/replaytest/consistency_report.go
Rememorio
left a comment
Rememorio
left a comment
There was a problem hiding this comment.
I reviewed the changed lines and left 6 inline comments below. The comments focus on issues worth addressing before merge.
Fresh review found one reliability issue and five test-harness gaps. Shared timestamp state is race-prone; concurrent writes are actually serialized; summary boundaries and event order are incompletely verified; malformed scenarios can panic; and the error-recovery case never exercises an error.
中文
我看过这次变更的相关代码，在下面留下 6 条行内评论。评论聚焦在合并前值得处理的问题。
本次审阅发现 1 个可靠性问题和 5 个测试框架缺口：共享时间状态存在竞态风险；所谓并发写入实际被串行化；摘要边界和事件顺序验证不完整；错误场景可能触发 panic；异常恢复用例也没有真正制造异常。
| // session creation (which uses wall-clock time.Now() inside the | ||
| // first step). This prevents SQLite's getSummariesList from | ||
| // discarding summaries. | ||
| stepBaseTime = time.Now().UTC().Truncate(time.Second) |
There was a problem hiding this comment.
P1: Keep replay timestamps out of package-global state
stepBaseTime is overwritten by every RunReplayCase call. Parallel subtests or backend runs will race and can change timestamps while another replay is still building events. Even sequential comparisons sample a separate base for each backend, so track timestamps can differ when the calls cross a one-second boundary.
Keep the clock local to a replay case, or inject one deterministic base shared by all compared backends, and pass it through event, track, and concurrent-step construction. A parallel race test and a clock-boundary test would cover both regressions.
中文
避免使用包级全局回放时间
stepBaseTime 是包级可变全局变量，每次调用 RunReplayCase 都会覆盖它。并行子测试或后端执行会产生数据竞争，还可能在另一个回放仍在构造事件时改变时间戳。即使顺序执行，两个后端也会分别取基准时间，跨过整秒边界时 track 时间戳会不同，导致用例偶发失败。
请将时钟作为单次回放的局部状态，或注入所有待比较后端共享的确定性基准时间，并传给事件、track 和并发步骤的构造逻辑。建议补充并行竞态测试和跨秒边界测试。
| case StepAppendEvent: | ||
| // Serialise GetSession + AppendEvent to avoid | ||
| // lost updates under CopyOnWrite semantics. | ||
| mu.Lock() |
There was a problem hiding this comment.
P2: The concurrent scenario serializes every backend write
This mutex covers the complete GetSession plus AppendEvent operation, and the same mutex also surrounds every memory operation. Only event preconstruction overlaps; all observable backend mutations are serialized. A backend that loses updates or races under concurrent AppendEvent calls would still pass, despite the claimed concurrent-write coverage.
Exercise genuinely overlapping service calls with a barrier and assert that every expected event survives. If this scenario is intentionally limited to nondeterministic ordering, rename it and adjust the documentation instead of presenting it as backend concurrency validation.
中文
并发场景实际串行化了所有后端写入
该互斥锁覆盖了完整的 GetSession 和 AppendEvent 操作，同一个锁还包住了所有 memory 操作。只有事件对象的预构造存在并行，所有可观察的后端写入仍是串行的。因此，即使某个后端在真正并发调用 AppendEvent 时丢失更新或发生竞态，这个用例仍会通过。
建议通过屏障让多个服务调用真正重叠，并断言所有预期事件都被保留。如果该场景只想验证非确定性顺序，应相应调整名称和文档，不要将其描述为后端并发写入验证。
| entry.Boundary = &replayBoundary{ | ||
| Version: boundary.Version, | ||
| FilterKey: boundary.FilterKey, | ||
| LastEventID: boundary.LastEventID, |
There was a problem hiding this comment.
P2: Summary cutoff corruption is omitted from comparison
replayBoundary defines CutoffAt, but this snapshot only copies Version, FilterKey, and LastEventID. A backend that persists the wrong cutoff therefore compares equal even though that timestamp controls which history is replayed. In addition, buildEvent leaves event IDs empty, so LastEventID never exercises real boundary anchoring.
Populate normalized CutoffAt using a deterministic shared clock, assign stable scenario event IDs, and assert all boundary fields in the summary-truncation case. That would detect the replay-corrupting boundary failures this case is intended to cover.
中文
摘要截止边界未纳入比较
replayBoundary 定义了 CutoffAt，但快照这里只复制了 Version、FilterKey 和 LastEventID。因此，后端即使错误持久化了截止时间，比较结果仍会相同，而该时间正是决定回放历史范围的关键字段。此外，buildEvent 没有设置事件 ID，导致 LastEventID 始终无法验证真实的边界锚点。
请使用共享的确定性时钟填充归一化后的 CutoffAt，为场景事件分配稳定 ID，并在摘要截断用例中断言全部边界字段。这样才能捕获该用例声称覆盖的回放边界错误。
| MemoriesCount *int `json:"memories_count,omitempty"` | ||
| NoDuplicateEvents bool `json:"no_duplicate_events,omitempty"` | ||
| NoDuplicateMemories bool `json:"no_duplicate_memories,omitempty"` | ||
| EventsOrderPreserved bool `json:"events_order_preserved,omitempty"` |
There was a problem hiding this comment.
P2: EventsOrderPreserved is never enforced
Four scenarios set events_order_preserved, but the field is never read. verifySnapshot checks only counts and duplicates, while cross-backend comparison proves only that the two backends agree. If both backends reorder the trajectory identically, the suite still passes.
Assert the snapshot against the scenario's append order using stable IDs or tags, and add a regression where both snapshots contain the same swapped order. Otherwise remove this verification option and the associated coverage claim until it is implemented.
中文
EventsOrderPreserved 从未被校验
四个场景设置了 events_order_preserved，但代码从未读取该字段。verifySnapshot 只检查数量和重复项，跨后端比较也只能证明两个后端结果一致。如果两个后端以相同方式错误地重排事件轨迹，测试仍会通过。
请利用稳定的事件 ID 或 tag，将快照顺序与场景中的追加顺序进行断言，并增加“两边都以相同方式交换顺序”的回归用例。否则应暂时移除该校验选项及相关覆盖声明。
| // validateSteps recursively validates step types, including steps nested | ||
| // inside Concurrent blocks. | ||
| func validateSteps(step ReplayStep, index int) error { | ||
| if !validStepTypes[step.Type] { |
There was a problem hiding this comment.
P2: Validate step payloads before execution
Validation checks only the step type. An append_event without event, a summary or track step without its payload, or a memory step with an invalid operation passes LoadReplayCase and later nil-dereferences or fails deep inside execution instead of returning a scenario error.
Validate the required payload and allowed operation for each step type, including permitted children of concurrent blocks. Table-driven malformed-case tests should assert descriptive load-time errors rather than panics.
中文
执行前应校验步骤载荷
当前校验只检查步骤类型。缺少 event 的 append_event、缺少载荷的 summary 或 track 步骤，以及操作值非法的 memory 步骤，都会通过 LoadReplayCase，随后在执行阶段发生空指针 panic 或产生难以定位的深层错误。
请针对每种步骤类型校验必需载荷和允许的操作值，同时限制并发块中的子步骤类型。建议使用表驱动测试确认错误场景在加载阶段返回清晰错误，而不是触发 panic。
| } | ||
| }, | ||
| { | ||
| "type": "append_event", |
There was a problem hiding this comment.
P2: The error-recovery case never injects an error
This is another successful append; the entire case performs four successful writes and only expects four events. No operation fails, no retry occurs, and no recovery behavior is asserted. A backend that commits and then returns an error, or duplicates an event during retry, would still pass this claimed coverage.
Use a fault-injecting service wrapper to fail before or after persistence, retry the operation, and assert the documented exactly-once or at-least-once behavior together with unchanged state.
中文
异常恢复用例没有制造异常
这里仍然只是一次成功的事件追加。整个用例执行了四次成功写入，并仅断言最终有四个事件；没有操作失败、没有重试，也没有验证任何恢复行为。如果后端写入成功后返回错误，或重试时重复插入事件，该用例依然无法发现。
建议使用可注入故障的服务包装器，分别模拟持久化前后失败，再执行重试，并根据约定断言 exactly-once 或 at-least-once 语义以及状态未被污染。
- P1: Keep replay timestamps out of package-global state — add BaseTime field to ReplayCase; RunReplayCase uses it when set (otherwise falls back to time.Now()); tests share a single baseTime across both backends so track timestamps are identical and never cross a second boundary - P2: True concurrent backend writes — replace global mutex with two-phase barrier (ready + start) in runConcurrentSteps; all goroutines released simultaneously call GetSession + AppendEvent independently; only aliases map uses a narrow lock; case_09 upgraded to 8 concurrent operations - P2: Summary cutoff corruption detection — add CutoffAtNonZero to summarySnapshot; remove CutoffAt from replayBoundary so wall-clock timestamps are verified for existence without false-diffing; assign explicit event IDs in scenarios so LastEventID is deterministic and cross-backend comparable - P2: Enforce EventsOrderPreserved — verifySnapshot now calls verifyEventsOrder using stable tags as event identifiers; greedy subsequence scan asserts snapshot events follow scenario append order; four scenarios that declare events_order_preserved are now actually verified - P2: Validate step payloads before execution — validateSteps checks required payload per step type (event, memory, summary, track); validates memory op values and non-empty concurrent blocks; table-driven TestReplayCase_Validate_RejectsMalformed with 15 cases catches malformation at load time - P2: Inject real errors in error-recovery — faultSessionService wrapper intercepts AppendEvent; FaultConfig supports fail_before (error before commit) and fail_after (commit then error); driver sets nextFault, tolerates expected errors and continues to retry step; case_10 rewritten accordingly - chore: blank line fix in consistency_report.go
|
Thanks for the thorough review! All issues have been addressed in 69d2b20 (and 88a9a91 for gofmt): |
3 participants
What changed
Introduced the
replaytestpackage — a replay consistency testingframework for Session and Memory services. The lightweight matrix
compares InMemory against SQLite with zero external dependencies.
Three design decisions worth calling out
SQLite Summary correctness:
buildEventpopulates Timestamp onboth
event.Event.TimestampandResponse.TimestampbecauseSummarizeSessionreads the top-level field. Omitting it causesSQLite's
getSummariesListto silently discard summaries via itscreatedAtfilter. All four summary failure modes (loss, contentcorruption, filter-key error, wrong session attribution) are verified
at 100% detection.
True concurrent event testing:
runConcurrentStepspre-buildsall events outside the critical section (pure functions, no
shared-state mutation), then locks only the GetSession+AppendEvent
window. All concurrent events share a single base timestamp to model
semantically simultaneous operations. Verified clean under
-race.Order-independent event comparison: instead of broad wildcard
exemptions that can mask field-level corruption, concurrent event
batches are sorted by deterministic JSON serialization before
comparison. This achieves field-level precision with zero
allowed_diffsrules across all 10 cases, while the mechanism itselfis retained as a safety hatch for future backend-specific gaps.
Scope
memory CRUD, summary generation, summary truncation with boundary,
track events, concurrent writes, error recovery
context locators (event_index, memory_id, summary_filter_key,
track_name)
TRPC_AGENT_REPLAY_REPORT_PATHoverridelightweight/integration modes, case format, report format
No runtime public API is changed.
Why
Different Session/Memory backends must behave identically when replaying
the same agent trajectory. Divergence in event order, state values,
memory contents, or summary metadata causes replay corruption that is
hard to debug without a dedicated harness.
Fixes #2001
Testing
Notes for reviewers
events_order_independent sorting uses encoding/json's deterministic map-key ordering, stable across Go versions.
allowed_diffs is intentionally unused in this PR; the mechanism exists for future backend-specific gaps (e.g., Redis TTL semantics).