How we run LongMemEval.

What LongMemEval is

LongMemEval (Wu et al., ICLR 2025) is a benchmark for long-conversation memory. Instead of a short prompt, the "haystack" is a stack of multi-session chat transcripts; each question's evidence is buried in an earlier session, and to answer it a system has to recall the right fact, notice when a fact was later updated, reason across sessions and over time, and — importantly — abstain when the memory genuinely doesn't contain the answer. LongMemEval_S is the roughly 500-question "small" split; it is the split we run. Higher is better.

How we run it

The suite runs against the real product — not a lab harness wired to a bespoke code path. Concretely:

• Through the real interface. The suite drives a FaultLine deployment over the same MCP interface a real connected AI uses — remember on ingest, recall on query. There is no benchmark-only shortcut around the memory engine. Runs happen on our pre-production environment, which runs the same build as production; we do not point the harness at customer infrastructure.
• Sharded across parallel seats. A dedicated, disposable benchmark account splits the LongMemEval_S set across several seats running in parallel, so a full pass completes without any seat's memory bleeding into another's — each seat is its own isolated tenant, which also exercises our per-tenant isolation under load.
• Against a fixed, hosted brain. Every seat is bound to the same cloud LLM (an Alibaba DashScope / QwenCloud endpoint). The key is held server-side and never handled by the harness. Binding a single, fixed, hosted brain is what keeps the model variable constant across runs, so a change reflects the memory engine and not a model swap.
• The stratified split. We run the stratified LongMemEval_S set, which keeps the mix of question types (single-session, multi-session, temporal-reasoning, knowledge-update, and abstention) balanced so no figure is dominated by the easy categories.

How we score it — deterministic, not LLM-judged

This is the honest part. Most published LongMemEval numbers are LLM-judged: a language model reads the system's answer and the gold answer and decides whether they "match." That grades leniently — a paraphrase, a hedge, or a lucky guess can be scored correct.

We score with a deterministic recall-containment check instead: the grounded fact the question needs must actually be present in what FaultLine returned, verified by a fixed, no-LLM rule. No judge model, no partial credit for vibes.

What our scorer cannot measure

A containment check is strict, and being strict has a cost we would rather state than bury. A deterministic rule only credits an answer whose gold text it can match against the recall, and a meaningful share of LongMemEval's questions are not of that shape.

• Abstention is not currently scored. The benchmark includes questions whose answer is deliberately absent from the haystack, where the correct behaviour is to decline. Our scorer has no abstention path: it compares gold text to recall text, and the gold answers for those questions are phrased as sentences about what was not provided. A correct refusal and a fabricated answer score identically — which means we currently have no measurement of abstention quality at all, in either direction. This is a limit of our instrument, not a result.
• Some answerable questions are unmatchable. Where a gold answer is a paraphrase or a reasoned restatement rather than a fact string, containment cannot credit a correct recall.

The consequence is that a deterministic aggregate over the full set is not comparable to an LLM-judged figure — it is not the same measurement made more strictly, it is a different and incomplete measurement. We are rebuilding the instrument before we put a headline number next to anyone else's.

Where the number comes out

We have not published a FaultLine score. We would rather show a blank than a figure we cannot defend — the same standard we apply below to competitors whose numbers we leave unclaimed. When the instrument measures what it claims to measure, the number will appear here with its run date and its method, and this page will say what changed.

Competitor numbers & sources

We cite competitors' own published LongMemEval figures, and we leave one unclaimed rather than invent or cherry-pick a number. These are the sources:

• Zep (Graphiti) — 63.8%. From the Zep paper, "Zep: A Temporal Knowledge Graph Architecture for Agent Memory" (arXiv:2501.13956), Table 2: 63.8% overall with GPT-4o-mini (and 71.2% with GPT-4o) versus a full-context baseline. This is an LLM-judged number. Source: arxiv.org/abs/2501.13956; vendor: getzep.com.
• Mem0 — unclaimed. Mem0's self-reported memory-benchmark figures and third-party measurements differ widely by configuration, benchmark variant, and judge model, so there is no single stable LongMemEval_S number we're comfortable attributing. We leave it blank with a link rather than pick the flattering one. Vendor / their benchmarks writeup: mem0.ai, mem0.ai/blog/ai-memory-benchmarks-in-2026.
• MemGPT / Letta — unclaimed. No vendor-published LongMemEval_S number located at time of writing. Vendor: letta.com.
• OpenAI Memory — unclaimed. OpenAI has not published a LongMemEval number for its assistant memory feature. Vendor: openai.com.
• The benchmark itself. LongMemEval, Wu et al., ICLR 2025 — arxiv.org/abs/2410.10813, code at github.com/xiaowu0162/LongMemEval.