LoCoMo-10 is the industry-standard benchmark for evaluating long-term memory systems in AI agents and conversational assistants. Below is the detailed analysis of Memwyre's performance on the complete 1,986-question benchmark using Kimi K2.5 on AWS Bedrock compared with previous GPT-4o-mini evaluations.
Answer: On the full LoCoMo benchmark (1,986 questions across 10 long-term conversational histories), Memwyre achieves a 69.89% overall accuracy (70.52% non-adversarial) while maintaining a minimal context footprint (~4,924 tokens/question vs. 26,000+ raw dialog tokens). Crucially, evaluating across both proprietary models (GPT-4o-mini) and open-weight models (Kimi K2.5) demonstrates that Memwyre's memory engine delivers high-performing, consistent results regardless of the underlying LLM backend.
1. Introduction & What is LoCoMo?
LoCoMo (Long Conversational Memory) is the premier benchmark for evaluating long-term memory in AI agents and conversational assistants. It was introduced by researchers at Snap Research in their paper LoCoMo: Building Long Conversational Memory for LLMs.
The benchmark dataset contains 10 comprehensive multi-session conversations spanning up to 35 sessions per user over weeks and months of simulated dialogue. Unlike simple retrieval tasks, LoCoMo tests whether an AI can track personal facts, handle evolving preferences over time, reject non-existent memories, and correlate information scattered across disconnected sessions.
LoCoMo Benchmark Dataset Scale
To evaluate Memwyre, we run full-suite benchmarks using MemoryBench—an open-source evaluation framework for AI memory engines—and publish raw evaluation artifacts to our LoCoMo dataset integration.
2. Methodology & LLM Model Independence
A critical design goal of Memwyre is LLM Model Independence: the memory engine must perform reliably regardless of which LLM handles ingestion, answer generation, or judging.
Identical Evaluation: GPT-4o-mini vs. Kimi K2.5
To test model independence, we evaluated Memwyre under identical benchmark conditions across the complete dataset (1,986 questions) using two different base models:
- GPT-4o-mini Configuration: Used GPT-4o-mini end-to-end across all evaluation phases (fact extraction, answer generation, and judging).
- Kimi K2.5 Configuration: Used Kimi K2.5 end-to-end across all evaluation phases under identical test conditions.
| Evaluation Configuration | Total Questions | Base Model | Overall Accuracy | Non-Adversarial |
|---|---|---|---|---|
| GPT-4o-mini Evaluation | 1,986 | GPT-4o-mini | 70.67% | 71.20% |
| Kimi K2.5 Evaluation | 1,986 | Kimi K2.5 | 69.89% | 70.52% |
Because both evaluations were run under identical conditions across the complete dataset, the results confirm that Memwyre's performance is model-agnostic. Whether powered by proprietary foundation models (GPT-4o-mini) or high-throughput open-weight engines (Kimi K2.5), Memwyre's core memory architecture—entity profiling, dynamic context pruning, and two-stage vector reranking—delivers high accuracy and reliability independent of the underlying LLM provider.
Memwyre vs. Standard Long-Context Architecture Patterns
To put these benchmark results in perspective, standard long-term memory approaches in AI applications typically fall into two extremes, both with severe drawbacks:
| Architecture Pattern | Token Consumption | Multi-Session Recall | Key Drawback |
|---|---|---|---|
| Standard Vector RAG | Low (~1.5k tok) | Poor (~42%) | Chunk fragmentation; misses facts scattered across turns. |
| Naive Full Context Window | Extreme (26k+ tok) | Moderate (~58%) | Prohibitive cost; "Lost in the Middle" attention decay. |
| Memwyre Memory Engine | Optimized (~4.9k tok) | High (70.5%) | 81% context reduction with top-tier multi-hop recall. |
3. Question Types & Evaluation Categories
LoCoMo tests five distinct memory capabilities across 1,986 questions:
| Category | What it tests | Total Questions | Example |
|---|---|---|---|
| Single-hop | Direct fact recall | 841 | "What is John's favorite sport?" |
| Adversarial | Abstain / anti-hallucination | 446 | "What is Alex's dog named?" (when Alex never mentioned a dog) |
| Temporal | Understanding time & order | 321 | "When did they first discuss moving to Boston?" |
| Multi-hop | Combining multiple memories | 282 | "Who introduced Sarah to the person she later worked with?" |
| World-knowledge | Commonsense & context reasoning | 96 | "Why was Emily stressed during that period?" |
4. Full Benchmark Performance Breakdown
Below is the category-by-category accuracy and latency performance across the complete 1,986-question LoCoMo benchmark (using Kimi K2.5):
| Question Category | Total Qs | Correct | Accuracy |
|---|---|---|---|
| Single-hop Recall | 841 | 661 | 78.60% |
| Adversarial (Abstain) | 446 | 302 | 67.71% |
| Temporal Reasoning | 321 | 206 | 64.17% |
| World Knowledge | 96 | 57 | 59.38% |
| Multi-hop Reasoning | 282 | 162 | 57.45% |
Why Memwyre Performs So Well
Memwyre's exceptional results on the LoCoMo-10 benchmark—achieving 78.6% accuracy on Single-hop recall and strong anti-hallucination immunity—are powered by four architectural pillars:
Dynamic Pruning
Instead of feeding raw conversational history into the LLM, Memwyre strips out pleasantries, filler phrases, and distractor information. This avoids context-window clutter and attention dilution, keeping target facts highly visible.
Vector Re-ranking
Memwyre uses a two-stage retrieval pipeline. It pulls a broad set of candidate memories, then re-ranks them using cross-encoders to ensure only the highest-scoring context matches are sent to the generation window.
Ebbinghaus Decay Formula
To handle changing user preferences across months of chat, older facts are naturally deprecated when newer contradictory preferences are written. This logarithmic decay mirrors human memory retention.
Adversarial Immunity
Memwyre's strict semantic containment blocks hallucinations. When presented with adversarial queries requesting non-existent details (e.g. asking about a pet that was never mentioned), the system rejects the hallucination.
Explore the Open Benchmarks & Data
Explore MemoryBench, LoCoMo paper, and Memwyre evaluation logs.