Definition
Plain language
A small chunk of training data processed at once when the full batch is too big to fit in memory.
As stated in the literature
A subdivision of a training batch processed sequentially under gradient accumulation; the locus of the DeepSpeed CPU-offload bug, where only the first micro-batch's gradients were copied to the optimizer.
Also called: micro-batches, microbatch
Why it matters: Handling these chunks incorrectly can silently corrupt training, as when only the first micro-batch's updates get applied and the rest are dropped.
For example, when a full training batch won't fit in memory, it's split into smaller micro-batches that are processed one after another.
Heard on the show
“Your ideal batch is bigger than your GPU memory allows, so you process the batch in chunks, called micro-batches.”Episode 009 — How Two Silent Library Bugs Quietly Invalidated a Wave of Reasoning Papers