Agent memory · LongMemEval · June 2026
Building and evaluating long-term conversational memory
We built three approaches to retaining information across conversations—raw retrieval, extracted memories, and a hybrid of both—and evaluated them on 500 LongMemEval questions. The hybrid approach reached 86.1% task-averaged accuracy.
Background
Why conversational agents use external memory
LLM-based assistants often need information from earlier conversations: user preferences, facts supplied by the assistant, or updates spread across multiple sessions. The simplest approach is to place the full conversation history in the model’s context window for every new request. As that history grows, prompts become longer, slower, and more expensive.
External memory offers another approach: store information from each session and retrieve only the context relevant to the current request. LongMemEval tests how well systems do this. Its Small split contains 500 questions over multi-session chat histories, covering user and assistant facts, preferences, knowledge updates, evidence spread across sessions, and temporal reasoning.
For each example, the memory strategy first ingests all prior conversation sessions and builds its memory store. We then submit the target test question, retrieve relevant memories, and add them to the answer prompt. The generated answer is compared with the reference answer by an LLM judge using LongMemEval’s official benchmarking scripts.
Memory strategies
Three approaches
Instruct
LLM-extracted memory
An extractor reads each session and the current memory store, then creates, updates, or deletes individual facts. This can consolidate changes across sessions, but details omitted during extraction are unavailable later.
Remis
Semantic retrieval from raw conversations
We call this strategy Remis. It splits sessions at topic shifts and indexes the resulting chunks with dense and BM25 signals. Neighboring chunks expand the retrieved context. This preserves exact details, but it can also return more irrelevant text.
Remis + Instruct
Raw excerpts + extracted facts
Both stores are queried, then one answer call receives two labeled sections: conversation excerpts and extracted memories. The answer model can draw from either source when responding.
Standalone Remis is described because it supplies the raw-conversation path in Remis + Instruct. The results focus on Instruct and the combined configuration.
Evaluation overview
What the evaluation measures
Accuracy. We report task-averaged accuracy across six LongMemEval question types.
Strategies. The main comparison is between Instruct and Remis + Instruct.
Cost. Estimates reflect observed model usage at list prices.
Results
Results for the tested strategies
The figures show overall accuracy, task-level results, and comparisons with other memory systems.
Complete runs
86.1% Remis + Instruct
71.2% Instruct
Higher accuracy across all six task types
The largest differences were on preferences and assistant facts. User facts were already strong with Instruct alone.
Accuracy by task type
LongMemEval small (%)
Task-averaged accuracy*
LongMemEval small (%)
Comparison systems
Remis + Instruct led the comparison
Remis + Instruct reached 86.1%, followed closely by Oracle at 86.0% and Mastra OM at 84.2%. Orange rows are systems we measured; gray rows are published reference values. The chart provides broad context rather than a controlled head-to-head ranking.
Why Remis + Instruct works
Extracted memories can consolidate information spread across sessions and represent changes over time. Raw chunks preserve quotes, names, dates, and numbers that an extractor may omit. Remis + Instruct gives the answer model access to both forms of context.
A July 2026 preprint, Beyond Memory Leaderboards, provides a useful comparison. In its scientific-retrieval evaluation, adding lexical BM25 to dense semantic retrieval was the largest intervention measured. Under a matched retrieval budget, the three resulting sparse–dense hybrids were effectively tied at the top of its PTr benchmark.
That design maps directly to the Remis path: Remis combines dense semantic search with BM25 over raw conversation chunks, while Instruct contributes separately extracted facts. Remis + Instruct therefore pairs sparse–dense raw-context retrieval with structured memory. The paper studies scientific documents rather than conversations, so it supports the retrieval pattern rather than validating our LongMemEval result.
Cost
Cost in context
We include systems for which our evaluation records contain both an accuracy result and enough token data to estimate visible LLM spend.
Accuracy and cost
Highest accuracy at about one-fifth of Mastra’s modeled cost
Among complete Small-split runs with fully observed model-token spend, Remis + Instruct scored 1.9 points above Mastra OM at $33.25 versus $171.65 per million conversation tokens. Instruct was cheaper but 14.9 points less accurate. Gray lower-bound points omit extraction or managed-ingestion spend, so their apparent cost advantage is incomplete.
Measured systems
Upper-left is better; cost axis is logarithmic
Reasoning-effort study
How reasoning effort affects extraction accuracy
We varied reasoning effort within each extraction model and measured the resulting accuracy and cost.
Reasoning effort, extraction accuracy, and cost
Higher accuracy and lower cost are better
Result
More reasoning did not always improve accuracy
More reasoning generally increased cost, but it did not guarantee better extraction. The best result came from GPT-5-mini at medium effort, the highest effort tested within that model family.
Summary
Raw evidence and extracted facts produced our strongest result
On LongMemEval Small, Remis + Instruct was the strongest configuration we measured: 86.1% task-averaged accuracy, with a higher score than Instruct in all six task types. The result supports combining hybrid retrieval over raw conversation chunks with extracted facts for long-term conversational memory.