Definition
Plain language
Pretending some of an AI's notes were stolen, and seeing how much performance drops.
As stated in the literature
In Auto-Dreamer, a counterfactual subset-masking procedure that randomly removes synthesized memory entries and measures the resulting performance change.
Why it matters: It distinguishes memory systems that genuinely contribute to performance from ones that look impressive but the model could ignore without losing anything.
For example, deleting a random 20% of an agent's synthesized memory entries and measuring how much its accuracy drops reveals how reliant it really is on those notes.
Heard on the show
“The framing in the brief I really like is the thief test.”Episode 064 — When Agent Memory Stops Being a Database and Starts Being a Skill