Definition
Plain language
Doing expensive AI work in advance, when no user is waiting, so the live response stays fast.
As stated in the literature
A family of techniques (including the contemporary Lin et al. work and the "Language Models Need Sleep" paper) that perform offline pre-processing — pre-generating likely queries or running depth-recurrent consolidation — between user requests to reduce inference latency.
Also called: sleep
Why it matters: It moves heavy thinking off the user-facing critical path, which can dramatically cut perceived latency without sacrificing answer quality.
For example, overnight a model pre-computes likely follow-up answers for tomorrow's users so morning queries return in milliseconds.
Heard on the show
“Then let's have it, because an implication that isn't about attackers is the one I'd actually lose sleep over.”Episode 243 — How a Hundred Meaningless Word Choices Add Up to Flip a Model's Answer