Benchmark methodology: how we measure 88.2% LoCoMo recall

[ferhimedamine]

What is LoCoMo?

LoCoMo (Long-term Conversation Memory) is an academic benchmark designed to evaluate how well AI systems retain and recall information from long multi-turn conversations. It was published by researchers at Meta AI and covers realistic conversation patterns: factual recall, temporal reasoning, preference tracking, and multi-hop questions.

We use LoCoMo as our primary quality gate because it tests the scenarios that matter most for agent memory:

• Can the system recall specific facts stated much earlier in a conversation?
• Can it reason about when something was said, not just what was said?
• Does it prefer recent context over older, lower-relevance memories?

Our current score: 88.2%

As of the latest stable release, Dakera achieves 88.2% recall accuracy on the full LoCoMo evaluation set. This score reflects our default configuration with no task-specific tuning.

How we run the evaluation

The benchmark is fully automated and runs in CI on every significant change to the recall pipeline:

# Clone the bench runner
git clone https://github.com/dakera-ai/dakera-deploy

# Run against a local Dakera instance
DAKERA_API_URL=http://localhost:3000 \
DAKERA_API_KEY=your-key \
./scripts/run-bench.sh

The runner:

1. Ingests the LoCoMo conversation dataset into a fresh Dakera instance
2. Issues each evaluation query against the memory store
3. Compares recalled content against ground-truth answers
4. Reports per-category accuracy and an overall score

What the score measures

The 88.2% figure is a recall accuracy — for each evaluation question, we check whether the correct memory is in the top-K retrieved results. We use the standard LoCoMo evaluation protocol so the number is directly comparable to published results from other memory systems.

What it doesn’t measure

LoCoMo is a retrieval benchmark, not an end-to-end agent benchmark. It tells you how well Dakera retrieves relevant memories, not how a downstream LLM uses those memories. Real-world performance also depends on your importance scoring strategy, your recall_k setting, and the quality of what you store.

Running it yourself

Want to verify the score on your own hardware or with a custom configuration?

# Start a local Dakera instance
cd docker && docker compose -f docker-compose.local.yml up -d

# Run the benchmark against it
DAKERA_API_URL=http://localhost:3000 ./scripts/run-bench.sh --full

Results will print per-category accuracy and an overall score. If you see a different number than 88.2%, we’d genuinely like to know — your hardware, dataset version, or configuration might surface an interesting edge case.

Transparency commitment

We report the score from a full benchmark run, not a cherry-picked subset. We don’t publish a score until the CI gate passes on the unmodified main branch. When the score changes (up or down), we note it in the release changelog.

---

Have questions about the evaluation methodology or want to compare notes on memory benchmarking? Drop them below.