Definition
Plain language
A kind of data or input — text, images, audio, or video — that a model can work with.
As stated in the literature
A distinct type of data stream (text, image, audio, video, action); models are called multimodal when they operate over more than one, and agent frameworks route each modality to a specialist that can handle it.
Also called: modalities
Why it matters: It defines what kinds of input a system can understand, so supporting more modalities lets an AI work with the world the way people do — through sight, sound, and words together.
For example, a photo is one modality and its spoken caption is another, and a model that can handle both is called multimodal.
Heard on the show
“The bet is: if the recipe works on text with zero modification, that's real evidence it's modality-agnostic.”Episode 127 — What Diffusion Language Models Were Missing: A Map, Not an Algorithm