Definition
Plain language
A benchmark for testing how well AI assistants remember information across long conversations.
As stated in the literature
A long-context memory evaluation suite testing retrieval and recall of user-provided facts across extended interaction histories; cited as part of the fact-recall paradigm that staleness-detection work argues is the easier half of agent memory.
Why it matters: It gauges whether long-term assistants can actually hold onto user facts, a basic requirement for feeling personal and reliable over time.
For example, it checks whether an assistant still remembers a user's stated dietary restriction many sessions after they mentioned it.
Heard on the show
“… Patel takes the knowledge-update questions from LongMemEval — a trusted benchmark where a fact is stated in one session and changed in a later one — and runs …”Episode 180 — The Bug Where Smart Assistants Read a Fact and Still Forget It