ML//multimodal

A multimodal system is an AI system that handles more than one modality (text, audio, image), and the systems that do so fall into three families that look alike from outside and differ in where the modalities meet.

A multimodal system is an AI system that handles more than one modality (text, audio, image), and the systems that do so fall into three families that look alike from outside and differ in where the modalities meet.

Diffusion models (Stable Diffusion, DALL-E 3, Midjourney) generate images but are architectures separate from the LLM. They do not reason with images; they produce them as output.

Native multimodal models (GPT-4o, Gemini) process and generate text, audio and image within the same architecture, a transformer whose attention runs over image and audio tokens as well as text, so the model reasons in several modalities simultaneously.

Hybrid pipelines use an LLM as the brain that orchestrates specialized models (one for text, one for image, one for voice). They look native, but inside they are pieces glued together, and their latency stacks, because inference passes through each specialized model.

Asking a native model to find the corroded valves in a photo needs no training for that request, only because the model already went through multimodal pretraining on vast numbers of image and text pairs (zero-shot learning): no training for the task, a great deal of it before.

An image is cut into patches that count as tokens, hundreds per image at typical resolutions, so image and video inputs cost more per request and take longer than text. A drone streaming frames to a remote model pays that on every frame, which is one reason vision often runs on the vehicle with a smaller model.