Definition
Plain language
A kind of data or input — text, images, audio, or video — that a model can work with.
As stated in the literature
A distinct type of data stream (text, image, audio, video, action); models are called multimodal when they operate over more than one, and agent frameworks route each modality to a specialist that can handle it.
Also called: modalities
Why it matters: It defines what kinds of input a system can understand, so supporting more modalities lets an AI work with the world the way people do — through sight, sound, and words together.
For example, a photo is one modality and its spoken caption is another, and a model that can handle both is called multimodal.
Heard on the show
“Freeze the world-model, allow only thin per-modality readouts, and see if text, image, and action all still work.”Episode 213 — A Model Learned to Control a Robot by Watching Video It Never Acted On