memory: add production-faithful LongMemEval benchmark

[Rememorio]

Summary

• add a production-faithful LongMemEval runner for pgvector main, pgvector candidate, and self-hosted Mem0 OSS
• replay sessions chronologically and trigger extraction after every user/assistant pair
• isolate every question and run while preserving the source observation date
• add stratified sampling, targeted replay, fixed-memory retrieval refresh, resumable answering and judging, bad-case analysis, and candidate comparison
• redact answer, evidence, memory, retrieval, and metric content from blind console and per-case logs while retaining sealed raw artifacts and operational usage\n- bind formal runs to a versioned preregistration manifest that verifies the exact selection, dataset, protocol payload, exclusion set, case metadata, and benchmark revision before provider setup
• record extraction operations, persistence diffs, retrieval hits, evidence recall, answers, errors, timings, token/cache/embedding usage, sanitized runtime configuration, and exact build provenance
• keep memory lineage pair-scoped so an answer-bearing extraction miss cannot mark a later unrelated memory as answer evidence
• aggregate independent answer and judge replicates into a machine-verifiable three-arm quality, category, error, usage, and cost promotion gate
• keep LoCoMo and LongMemEval logic in dataset-specific files and share protocol-level evaluation helpers
• keep the result index and unified English/Chinese reports aligned with the current protocol-v2 evidence

Fairness and diagnostics

All three arms receive the same dated user/assistant pairs, answer model and prompt, semantic judge protocol, Top-K 30 search limit, and isolated identity scopes. Pgvector uses the production MemoryService.EnqueueAutoMemoryJob path and waits for each job before replay continues. Self-hosted Mem0 uses its public OSS API and configured open-source update behavior. Provider calls whose responses omit usage remain counted through usage_missing_calls instead of being treated as zero-cost calls.

The benchmark pins both the trpc-agent-go root module and independently versioned memory/pgvector module to the current bd6b31f92a90 candidate. The evaluated compact-recovery tree eaf5f49f1fa4 is byte-equivalent to that PR head. Manifests record question IDs, replay counts, model variants, prompt versions, dependency replacements, table identities, cache hashes, snapshot hashes, source revisions, image digests, and usage coverage.

Selection-only mode writes the full versioned protocol payload and digest. A preregistered run rejects ad hoc filters, resampling, truncation, dataset or protocol drift, exclusion-set drift, case-metadata drift, dirty builds, and benchmark-revision drift before model or provider initialization. Frozen exclusions are validated against the active dataset and represented by a canonical count and digest without retaining a local path.

Traces separate extraction misses, reconciliation or persistence losses, retrieval misses, answer failures, and judge failures. Pair-scoped provenance prevents a later unrelated memory from being credited for an answer-bearing pair that produced no memory. Recovery traces retain initial and recovered answers, the selected source, and whether recovery was applied. Formal answer replicates use empty, independent answer and judge cache ledgers while sharing immutable ingestion and retrieval snapshots.

Current evidence

The formal protocol-v2 comparison uses one fixed, already observed 16-question development set spanning all six LongMemEval types. Each arm ingested the complete selected haystack: 183 dated conversation pairs, no session or pair truncation, Top-K 30, three independent answers per question, and three independent judge votes per answer.

Arm	Primary	Majority	Correct answer replicates	Unstable cases	Pipeline errors
pgvector upstream main	12/16	12/16	36/48	0	0
self-hosted Mem0 OSS	14/16	14/16	41/48	1	0
pgvector candidate	16/16	16/16	48/48	0	0

The candidate had no category deficit on this set: knowledge-update 2/2, multi-session 4/4, assistant 2/2, preference 2/2, user 3/3, and temporal 3/3. Main's stable misses include two assistant-result cases, one preference-history case, and one answer-context case where the required memory was already stored and retrieved. Mem0's stable failures expose assistant-result provenance and retrieval-context selection issues; its additional unstable case scored 2/3. These distinctions are retained in the traces so answer behavior is not mislabeled as extraction or retrieval quality.

The final compact-recovery candidate preserves 48/48 correct answers and 144/144 correct judge votes. Relative to its parent candidate, recovery calls fall from 27 to 24, memory LLM calls from 210 to 207, memory LLM tokens from 1,838,020 to 1,790,001 (-2.6%), and embedding tokens by 2.6%. Final memories increase from 410 to 416 and observed ingestion time increases from 1,753s to 2,301s, so this is not presented as a storage or latency improvement. Removing recovery entirely was rejected after the assistant-only e3fc4d6e case produced no memory or retrieval evidence.

Relative to main, final-candidate memory-layer LLM tokens are 1.507x, embedding tokens 1.639x, and persisted memories 2.755x. Relative to Mem0, candidate memory-layer LLM tokens increase 1.4%, embedding tokens decrease 69.2%, and persisted memories decrease 14.2%. Answer and judge usage is reported separately because cache behavior makes it unsuitable for a memory-layer cost comparison.

An immutable 10-question LoCoMo regression sample, stratified as two questions per category with three QA repeats, scored 0.6825 mean F1 for candidate versus 0.7107 for the previous focused-passage control. The -0.0282 delta passes the preregistered -0.05 bound but is reported as bounded regression, not an improvement. All of the difference is concentrated in two questions: one answer recovery omitted a supported clause, and two recovery calls for a long list exhausted their token limit; the other eight questions have identical three-repeat means.

The formal run and compact ablation passed checksum, isolation, provenance, usage-completeness, replication, category, error, and cost audits. Exact digests and integrated interpretation are in the English report and Chinese report. Earlier seed48 and seed137 experiments remain only as historical protocol-v1 diagnostics. No new unseen blind holdout has been run or selected for this candidate.

Agent dependency

• root module: github.com/Rememorio/trpc-agent-go@v0.0.0-20260721030639-bd6b31f92a90
• pgvector module: github.com/Rememorio/trpc-agent-go/memory/pgvector@v0.0.0-20260721030639-bd6b31f92a90
• upstream draft: memory: improve automatic memory extraction and retrieval trpc-agent-go#2196

Validation

• go test ./... -count=1, go test -race ./... -count=1, and go vet ./... in memory/trpc-agent-go-impl
• synthetic three-arm gate tests covering retrieval drift, cache-ledger isolation, missing usage, invalid vote counts, malformed manifests, and resource-bound failures
• clean-revision runner smoke covering full-haystack Top-K 30 preregistration and pre-provider rejection of protocol, exclusion, and case-set drift
• self-hosted Mem0 add, search, answer, and judge smoke with complete provider usage
• audited protocol-v2 dev16, compact-recovery ablation, and LoCoMo artifacts with immutable snapshots, exact provenance, atomic finalization, and verified checksums