Definition
Plain language
How long you wait between asking an AI and seeing the first word of its reply.
As stated in the literature
A serving latency metric measuring the delay before the first output token is produced; in agentic and long-context workloads it can spike badly at the tail (P99) when KV-cache management and scheduling break down.
Also called: time to first token, TTFT
Why it matters: It strongly shapes how responsive an AI feels, and long waits at the tail can make an otherwise capable system feel sluggish and frustrating to use.
For example, you type a question and stare at a blank screen for two seconds before the first word appears — that wait is the time-to-first-token.
Heard on the show
“The paper measures time-to-first-token for each round of an agent loop, and reports that baseline systems consistently exceed a thousand seconds at the P99 tail.”Episode 016 — Why Your Coding Agent Stalls While the GPU Runs Hot