Definition
Plain language
How long you wait between asking a computer for something and getting the response.
As stated in the literature
The time delay between a request and its response; in inference, dominated by time-to-first-token and per-token generation time.
Why it matters: High latency makes a tool feel sluggish and unusable for anything interactive or real-time, so keeping it low is essential for good experiences.
For example, when you type a question and the answer appears almost instantly, the tiny wait you notice is the latency.
Heard on the show
“… from real public leaderboard percentiles, and the speed score from each provider's own published latency numbers, so the page read as legitimate rather than as an obvious plant. …”Episode 205 — The Same AI, Two Labels: How the Pitch Beat the Product in 162 Sessions