ML//multimodal//CLIP

CLIP (Contrastive Language-Image Pre-training, OpenAI, 2021) is a pair of neural encoders, one for images and one for text, trained together so that matching image and caption pairs land at nearby points in a shared space. It is the bridge between vision and language, and it powers Stable Diffusion's text conditioning.

CLIP (Contrastive Language-Image Pre-training, OpenAI, 2021) is a pair of neural encoders, one for images and one for text, trained together so that matching image and caption pairs land at nearby points in a shared space. It is the bridge between vision and language, and it powers Stable Diffusion's text conditioning.

It allows zero-shot classification: the text a photo of a cat matches cat images with no cat-specific training.