Definition
Plain language
A way of running a model server that lets new requests jump into the batch the moment others finish, instead of waiting for the whole batch to complete.
As stated in the literature
An LLM serving technique that dynamically adds and removes requests from an in-flight batch at the token level, keeping GPU utilization high under variable-length generations; a core throughput optimization in stacks like vLLM, and one reason a long generation doesn't cost wall-clock time as linearly as a naive token count implies.
Why it matters: It keeps expensive hardware busy and lets a model server handle far more simultaneous users without making everyone wait for the slowest request.
For example, when one user's short request finishes early, the server immediately slots a waiting request into the freed-up space instead of letting the hardware sit idle.