In agent-memory research, two benchmarks are virtually inescapable — everyone runs them: LoCoMo and LongMemEval. But does a high score on either truly prove anything? Are there more targeted benchmarks better suited to specific use cases? Should memory evaluation be reduced to accuracy alone? What other metrics matter, and how are the data synthesis pipelines behind these benchmarks actually constructed?

These are the questions this article sets out to answer.

Overall

Agent-memory benchmarks broadly fall into two categories: conversational memory and agent task improvement. The former concerns whether the system can remember important user information; the latter focuses on whether the agent can reduce similar types of errors over time.

Naturally, the evaluation approaches differ — for task improvement, standard coding benchmarks can often serve the purpose. This article focuses on the former: conversational memory.

Memory benchmarks can be understood along two orthogonal axes.

Axis 1: What content / scenario is being tested (horizontal) — factual conversational memory (LoCoMo / LongMemEval), personalization & user profiling (PersonaMem / PersonaLens), experiential / environmental memory (LongMemEval-V2), and multimodal memory.

Axis 2: What capability dimension is being tested (vertical) — lifecycle abilities (update, conflict, forgetting, abstention, temporal reasoning). These cut across all scenarios.

Long-term Conversational Memory

The earliest and most frequently evaluated category of personal memory is long-term conversational memory: given dialogues spanning multiple turns and multiple sessions, can the agent remember user-relevant information and correctly recall it in subsequent questions or responses?

This type of benchmark closely resembles traditional long-context QA. The emphasis is on whether the agent can locate, understand, and apply user-related memories across very long conversation histories — both within long sessions and across sessions.

LoCoMo

📄 LoCoMo Paper · GitHub Repository

LoCoMo (Long Conversation Memory) is one of the most frequently cited benchmarks in agent-memory research.

LoCoMo’s data is generated through a hybrid human-machine pipeline: starting from predefined personas and temporal event graphs, two LLM agents generate long conversation skeletons, which human annotators then verify and edit to ensure long-range consistency, event grounding, and multimodal content plausibility. The public release contains 10 ultra-long conversations, each averaging roughly 300 turns and 9K tokens, spanning up to 35 sessions.

LoCoMo defines three evaluation tasks.

The first task is Question Answering. This is the part most commonly used by subsequent memory system papers. QA questions fall into five categories:

The second task is Event Summarization. The model must summarize the conversation history into a temporal event graph, testing whether it understands events, causal relationships, and temporal ordering in long-term dialogues.

The third task is Multimodal Dialogue Generation. LoCoMo’s dialogues include image sharing and image reactions, so this task evaluates whether the model can combine past dialogue and multimodal cues to generate coherent responses consistent with character state and long-term narrative.

From a data structure perspective (locomo10.json), each element in the JSON array represents one complete long conversation with its annotations.

Evaluation methodology and metrics. Evaluation is not LLM-Judge-dominated.

• QA: normalized / partial-match F1; RAG systems additionally report retrieval recall / recall@k.
• Event Summarization: ROUGE + FactScore, where FactScore decomposes into atomic facts and computes precision / recall / F1.
• Multimodal Dialogue Generation: BLEU-1/2, ROUGE-L, MM-Relevance.

Its limitations, however, are clear. At roughly 9K tokens on average, the public version is no longer sufficient in 2026 to meaningfully differentiate long-context models. Furthermore, most follow-up work focuses primarily on LoCoMo’s QA component, reducing it to a text retrieval / QA benchmark; the event summarization and multimodal dialogue generation tasks remain underutilized in practical system comparisons.

This has led to targeted corrections such as LoCoMo-Plus and LoCoMo-Refined. LoCoMo-Plus’s direction is particularly noteworthy: it attempts to move from surface-level factual recall to cognitive memory — testing whether a model can apply implicit user states, goals, values, and constraints from earlier dialogues in subsequent scenarios.

📄 LoCoMo-Plus Paper · LoCoMo-Refined GitHub Repository

LongMemEval

Compared to LoCoMo, LongMemEval is more directly oriented toward long-term interaction memory for chat assistants. Its goal is to evaluate whether an assistant can record, retrieve, update, and reason about user information across sustained, multi-session user interactions.

📄 LongMemEval Paper · GitHub Repository

LongMemEval contains 500 questions covering five categories of long-term memory capability:

LongMemEval offers different difficulty tiers. LongMemEval-S provides roughly 115K tokens of history per question, typically spanning 30–40 sessions; LongMemEval-M scales to roughly 500 sessions and 1.5M tokens. This makes it far more suitable than LoCoMo for testing the memory scalability of real personal assistants over extended use.

For evaluation, LongMemEval employs LLM-as-a-Judge, with GPT-4o judging whether a model’s answer is correct. It also proposes a useful memory system analysis framework: decomposing a long-term memory system into three stages — indexing, retrieval, and reading.

From a personal memory perspective, LongMemEval is closer to real user-assistant scenarios than LoCoMo. It tests not only “does it remember,” but also whether user information updates over time, whether information can be synthesized across sessions, whether temporal reasoning is handled correctly, and whether the system refuses to answer when information is absent.

That said, LongMemEval remains fundamentally a long-context QA / retrieval-oriented benchmark. Its core evaluation format still revolves around finding evidence in chat histories and answering questions. But real-world agents face a harder problem: state is dynamic, making the benchmark susceptible to retrieval engineering.

From LongMemEval to LongMemEval-V2

📄 LongMemEval-V2 GitHub Repository

No longer merely testing “can a chat assistant remember long-term conversations,” but rather “can an agent form long-term experiential memory in complex environments.”

V2’s object of study is no longer ordinary user-assistant chat history, but web-agent trajectories, corresponding to five categories of memory ability:

V2’s context scale explodes:

• small tier: 25M tokens
• medium tier: 115M tokens

This far exceeds mainstream context windows.

This represents arguably one of the biggest directional shifts in AI memory research over the past year.

“How can LLMs remember long-term conversations” → “How can agents accumulate long-term work experience”

Or put differently: chatbot memory → agentic / experiential memory.

From a data synthesis pipeline perspective:

LongMemEval is constructed by first designing the user memories to be tested, then embedding those memories into long chat histories. It roughly follows three steps:

• question curation: An ontology of 164 user attributes (e.g., lifestyle, belongings) is defined. An LLM generates user background paragraphs for each attribute, from which candidate QA pairs are generated (with human filtering, rewriting, and difficulty calibration). Annotators decompose answers into one or more evidence statements (some carrying temporal information).
• evidence session construction: Each evidence statement is embedded into a task-oriented user-assistant session, using self-chat to generate dialogue. The user does not explicitly state the evidence; instead, information is revealed indirectly through requests for help, advice, or task completion.
• history compilation: Relevant evidence sessions are inserted into a longer user-assistant chat history. Irrelevant sessions come from two sources: self-chat sessions synthesized from other non-conflicting user attributes, and ShareGPT + UltraChat. All sessions are assigned reasonable timestamps, producing a coherent, length-configurable long-term chat history.

At its core, this remains needle-in-a-haystack for personal memory.

By contrast, LongMemEval-V2 is constructed by first collecting agent trajectories from task execution in environments, then extracting from those trajectories the knowledge an experienced agent ought to remember.

It roughly follows four steps:

• trajectory collection: Web-agent trajectories are collected from WebArena, WorkArena, and WorkArena++ via the AgentLab unified interface. Each trajectory includes task goal, state observations, screenshots, accessibility tree, actions, and state transitions. Both successful and failed trajectories are collected through rejection sampling.
• question annotation: Questions are constructed around five memory ability categories: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. The model is tested on whether it can answer candidate questions without trajectory evidence — if it can, the questions are filtered or rewritten (preventing the model from answering via public knowledge alone).
• answer trajectory labeling: Each question requires labeling which trajectories genuinely contain answer evidence, with the requirement that evidence be verifiable in screenshots or trajectory states (a batch of seed answer-bearing trajectories is used with Codex to expand the candidate annotation pool, followed by human verification of ambiguous cases).
• haystack creation: The Small tier constructs one shared 100-trajectory haystack each for WebArena and ServiceNow; the Medium tier constructs roughly 500-trajectory question-specific haystacks for each question.

EverMemBench

📄 EverMemBench Paper

From dyadic conversations to multi-person collaboration. Can an agent embedded in long-term team collaboration maintain memory across multiple people, groups, and timelines?

EverMemBench simulates enterprise collaboration scenarios. The data includes multiple projects, sub-projects, employees, group conversations, and evidence-grounded QA. The overall structure resembles:

project
  ├── sub-project / group chat
  │     ├── day 1 messages
  │     ├── day 2 messages
  │     └── ...
  ├── employees / personas
  ├── task timeline
  └── QA items with evidence references

Evaluation pipeline:

Add → Search → Answer → Evaluate

EverMemBench’s QA is organized across three dimensions and nine subtasks:

EverMemBench’s evaluation approach can be divided into two layers: task protocol and scoring method.

For task protocol, it uses a streaming multi-group protocol: during the ingestion phase, the system receives daily batches in chronological order and must maintain its own memory state; during the evaluation phase, questions are posed. This protocol tests how the system writes, updates, retrieves, and uses memory while continuously receiving information, rather than dumping all history into context at once.

For scoring:

Data construction can be summarized as:

blueprint and profile generation
→ chunk-wise dialogue synthesis
→ evidence-grounded QA construction

User Profiling and Personalization

PersonaMem

📄 PersonaMem Paper · GitHub Repository

The core question is not “can it recite a particular fact,” but: can the model form and update user profiles across long-range, multi-turn, multi-session user interaction histories, and use the most current user state to generate personalized responses?

Three capabilities: whether the model can internalize users’ intrinsic characteristics and preferences; whether it can track the temporal evolution of user profiles / preferences; and whether it can generate personalized responses in new scenarios accordingly.

The setup: given the user-LLM interaction history up to a certain point in time, plus a first-person in-situ user query, the model must select the response that best matches the current user state.

It tests four things:

Remembering facts / preferences users have shared: e.g., a user once expressed a preference for a certain type of music, diet, or learning style.

Updating memories: Preferences change — the model must not cling to old preferences, e.g., shifting from liking pizza to going gluten-free due to a gluten allergy.

Retrieving relevant fragments from history: Information may appear far back in long histories, with “lost in the middle” problems.

Applying memories to responses, not merely parroting: The hardest part is giving new recommendations based on current preferences, or transferring to new scenarios. V1 found models perform reasonably on simple factual recall, but are significantly weaker at “giving new suggestions” and “applying user preferences across scenarios.”

V1 data scale: roughly 20 personas, 180+ interaction histories, each history up to 60 multi-turn sessions, covering 15 personalized task scenarios; the HF dataset provides 32K, 128K, and 1M context-length versions.

It defines 7 types of in-situ user queries to cover different memory / personalization skills:

V1’s primary evaluation is 4-way multiple-choice accuracy: the model sees the full conversation history, the current user query, and four candidate responses, and selects the response that best matches the current user profile.

V2 improvements:

V2’s positioning reflects a clear pivot: from “explicit/dynamic preference memory” to “implicit preference inference + trainable agentic memory.”

V2 particularly focuses on cases where user preferences are not stated directly but are embedded in long dialogues — for instance, when a user asks the model to polish an email, the email content may inadvertently reveal seasonal allergies.

Each benchmark row in V2 includes: persona_id, path to the full persona file, links to 32K/128K chat history, user_query, correct_answer, three incorrect_answers, current preference, preference category, relevant dialogue snippets, whether the preference belongs to the user, whether it has been updated, old preference, sensitive information, and token distance from relevant snippet to query.

🤗 PersonaMem-v2 Dataset

V2’s primary evaluation metrics:

1. MCQ accuracy: The model selects the most personalized answer from four plausible options.
2. Open-ended personalization accuracy / judge score: The model generates a free-form response, then LLM-as-a-judge determines whether it reflects the ground-truth preference; the paper reports aggregation from multiple GPT-5-Chat judges.
3. Grouped analysis: Performance is broken down by preference type, scenario, whether the preference was dynamically updated, who, distance to relevant snippet, and other dimensions. The official evaluation scripts also generate these category breakdowns.

PersonaLens

Where PersonaMem treats personalization as a memory QA problem, PersonaLens is more concerned with whether personalization gets squeezed out by task objectives during task completion.

📄 PersonaLens Paper

In terms of composition, it has 1,500 user profiles (demographic attributes drawn from the PRISM Alignment dataset, with preferences generated by an LLM based on demographics to ensure internal consistency), covering roughly 20 domains from Books, Finance, Flights to Restaurants.

Evaluation methodology: dual-agent automatic evaluation — a user agent (implemented with Claude 3 Sonnet) simulates a user with a specific profile and context conversing with the assistant under test; a judge agent uses LLM-as-a-Judge to rate personalization, response quality, and task completion, distinguishing between single-domain (TSD) and multi-domain (TMD) tasks.

A key finding: there exists a trade-off between task completion rate (TCR) and personalization — the more the assistant buries its head in getting the task done, the more likely it is to ignore user preferences.

By contrast, static QA has no “task objective” competing with personalization for attention.

Multimodal Personal Memory

Memory originates as text. But real personal memory is clearly more than that.

Can an agent build long-term memory across text, images, and video, and then invoke those memories in downstream tasks?

In much of what is called multimodal memory, images are merely captioned into text and fed back into the text pipeline. LoCoMo itself includes a multimodal component (image sharing), but the vast majority of subsequent work has reduced it to pure text QA.

A useful way to think about this is to ask: at which stage does multimodality actually play a role?

Input / storage side: preserving visual information across sessions.

A user shares an image in an earlier session; the system must remember its content later.

📄 Mem-Gallery Paper

Mem-Gallery explicitly points out that existing benchmarks either evaluate memory in purely textual multi-session settings, or evaluate multimodal understanding in localized single-session contexts — none genuinely assess how multimodal memory is preserved, organized, and evolved over long dialogue trajectories. It evaluates along three dimensions: memory extraction & test-time adaptation, memory reasoning, and memory knowledge management.

Retrieval side: cross-modal retrieval within visual history. Storage is only the first step; whether the right image / frame can be found based on the current need is another matter entirely.

Layered with personalization: multimodal × user modeling.

Persona-MME targets long-term personalized multimodal LLMs by constructing persistent multimodal persona contexts (user profile, frequently occurring social relationships, objects the user owns) and then generating three types of multimodal dialogue — explicit entities, implicit facts, and distractors — to control the memory signals introduced.

Layered with real / embodied scenarios. This falls within the embodied / world model direction: egocentric wearable assistants, robot memory, video world model memory capabilities, etc. (Beyond the scope of this article.)

On the positive side, Mem-Gallery, MemLens, and similar work from early 2026 represent a correction to the “images get captioned away” problem.

But this category as a whole is still very young. It has not yet converged on a universally-adopted benchmark like LoCoMo; modality coverage, task formats, and evaluation standards remain inconsistent across efforts.

Memory Lifecycle Capabilities

The preceding sections were organized by “what content is being tested”: conversational facts, user profiles, multimodal.

But can the agent update old memories? Handle conflicting information? Know when to forget or abstain? Track how preferences change over time? These are the lifecycle capabilities of memory.

The lifecycle landscape is fragmented — a collection of specialized benchmarks each targeting a single capability, most emerging only from late 2025 through 2026. This is itself a signal: the field is only just realizing that accuracy alone is not enough. Specialized evaluations are beginning to appear but have not yet converged.

Below, organized by capability dimension, with representative benchmarks and the metrics — distinct from accuracy — that each introduces.

Update

A memory system must remember the most recent version, not the first thing that was said.

LongMemEval already includes a dedicated knowledge-update subset (described earlier). Going further, some work has turned this into a metric rather than just a data split: Forgetting-Aware Memory Accuracy specifically penalizes reliance on stale or invalidated memories.

Conflict Resolution

When two contradictory pieces of information coexist in memory, can the system ensure only the latest, correct one takes effect?

MemoryAgentBench addresses this: decomposing memory capability into accurate retrieval, test-time learning, long-range comprehension, and conflict resolution, with a dedicated Conflict Resolution Accuracy metric ensuring that when contradictions exist, only the most recent valid fact is used.

Forgetting / Selective Retention

Not all information should be kept forever: expired information and anything the user explicitly asks “don’t remember this” should be discarded.

MemoryAgentBench uses its newly constructed FactConsolidation dataset to specifically test selective forgetting.

User-initiated forgetting is also tied to privacy, as in PersonaMem-v2’s privacy scenarios (user requests not to remember a certain preference).

Abstention

If you don’t know, say you don’t know. A memory system must recognize “insufficient evidence” and abstain, rather than hallucinate.

LongMemEval specifically includes abstention-type questions to test this, with corresponding metrics of abstention rate / abstention accuracy — again, not captured by standard accuracy.

From Passive Recall to Memory-Driven Action

MemoryArena shows that models scoring near-perfect on recall benchmarks plummet in performance the moment memory must drive action. Many so-called memory benchmarks are essentially retrieval benchmarks, testing “can the needle be found” rather than “can memory actually be put to use.”

Evaluation Metrics: Beyond Accuracy

If we only look at accuracy, what gets missed?

Borrowing an existing layered framework, a recent memory survey proposed a four-layer evaluation stack that serves as a useful skeleton for this section.

Layer 1: Task Effectiveness

Did the task actually get done?

Common metrics include success rate, factual correctness, plan completion rate. In the QA format typical of conversational memory benchmarks, these are the familiar: token-level F1, BLEU, binary/rating judgments from LLM-as-a-Judge, and MCQ accuracy.

These metrics are not equivalent to one another — which one you choose changes the conclusion. Even though LLM-as-a-Judge is closer to semantic correctness, it introduces the judge’s own biases.

At this point, “who is SOTA” depends on which metric you trust.

Layer 2: Memory Quality

Key metrics include: precision / recall of retrieved records, contradiction rate, staleness distribution, and coverage of task-relevant facts.

Some work on LongMemEval-S has achieved nearly 99% retrieval recall for certain question types using retrieval alone — but that’s retrieval recall, not end-to-end accuracy.

Staleness distribution and contradiction rate focus on “how much of the stored memory has expired or is internally contradictory.”

Correspondingly, specialized metrics have emerged, such as Forgetting-Aware Memory Accuracy (penalizing reliance on stale memories) and conflict resolution accuracy (ensuring only the latest version of contradictory information takes effect).

Layer 3: Efficiency

Core metrics: latency per memory operation, prompt tokens consumed by memory content, retrieval call count per step, and storage growth over time.

A methodological detail worth noting: rigorous work (e.g., HiMem) reports latency by measuring only the time spent on memory retrieval itself, excluding LLM inference, to isolate memory system efficiency and avoid contamination by the speed of the underlying model. When reading efficiency numbers, pay attention to which segment is actually being measured.

Layer 4: Governance

Does the memory system comply with privacy and control requirements?

Metrics include: privacy leakage rate, deletion compliance (when the user says “don’t remember / delete this,” does the system actually comply), and access-scope violations (unauthorized access, especially in multi-user shared memory scenarios).

PersonaMem-v2 already incorporates such scenarios: users leaking sensitive information, or explicitly requesting “don’t remember this preference,” with target answers required to avoid using such sensitive / private information.

These four layers are often in tension with one another.

A thorough system evaluation should therefore present an operating curve across the four layers (a tunable trade-off among accuracy / store-size / latency), rather than a single isolated best score.

Data Synthesis Methods

Before discussing any memory benchmark, a more fundamental question should be asked: how were these multi-turn dialogues constructed? Because the synthesis method almost entirely determines what a benchmark actually tests — and what it is inherently incapable of testing.

Current mainstream benchmark dialogue sources fall roughly into three tiers:

• Purely human-authored real data (virtually nonexistent)
• LLM-generated skeleton + human verification (exemplified by LoCoMo: based on predefined personas and temporal event graphs, dialogue skeletons are generated using an LLM agent architecture, then human annotators verify and edit for long-range consistency)
• Needle-in-a-haystack assembly (exemplified by LongMemEval: questions written by human experts, evidence-bearing sessions simulated by LLMs and manually edited, while the vast majority of irrelevant sessions are sampled from ShareGPT, UltraChat, and simulated dialogues)

Personalization-oriented benchmarks rely more heavily on persona-driven fully-automated synthesis. PersonaMem uses a GPT-4o-driven modular pipeline to generate multi-session, temporally-structured interaction histories from personas, embedding preferences that evolve over time within the dialogues; its in-situ query appends the user’s current first-person question to the end of the history, testing whether “the model can answer using the most recent preferences.”

Since needle-in-a-haystack is fundamentally a needle-finding test, what it predominantly measures is retrieval capability within long contexts, rather than the organizational and evolutionary capabilities of a memory system.

When LongMemEval selects haystack sessions, it deliberately chooses sessions similar in topic and format to the evidence sessions, but without introducing conflicting information, so as not to invalidate the questions. This engineering-sound choice means that this benchmark, from its construction starting point, deliberately avoids knowledge conflicts — and is therefore inherently incapable of testing conflict resolution.

As for the common concern about “whether LLM-generated data is trustworthy”: the answer lies not in LLM hallucination per se, but in the filtering and verification after generation. PersonaMem-v2, for example, applies four hard filters to every Q&A pair — the question must be unanswerable without the history (preventing data leakage and generation artifacts), the correct option must genuinely reflect user preferences, incorrect options must never reflect them, and the format must not retain generation traces. Ultimately, only roughly 30% of generated Q&A pairs pass all filters.

The pass rate itself reveals that, without rigorous filtering, a significant proportion of synthetic dialogues contain shortcuts exploitable by models.

Thus, the synthesis pipeline itself defines the boundaries of what a benchmark can test.

Closing

No single benchmark can cover everything that agent-memory entails. Every facet — factual recall, personalized inference, multimodality, update, conflict, forgetting, abstention, memory-driven action — requires purpose-built data and purpose-designed metrics.

So when an agent-memory project claims yet another SOTA, the set of questions worth asking remains the same: SOTA on which benchmark? What core capability does that benchmark measure, and what properties does it miss? What does its data construction allow it to test, and what is it structurally unable to test? Which layers of metrics does it report, and are the missing layers precisely the ones it sacrificed?