Definition
Plain language
Running a finished AI model to get an answer, as opposed to training it in the first place.
As stated in the literature
The forward-pass execution of a trained model to produce outputs, distinct from training; the regime where serving cost, latency, KV-cache memory, and test-time scaling techniques live.
Also called: inference-time, inference time
Why it matters: It is where the real-world costs of running AI live, since speed, memory use, and serving expense all determine whether a model is practical to deploy.
For example, every time you type a question into a chatbot and get an answer, that's inference, separate from the earlier training that built the model.
Heard on the show
“These are six to sixteen percentage points of accuracy, on top of frontier models, with no training, just clever inference.”Episode 003 — How to Pick the Best of Sixteen Coding Agent Rollouts