ML//multimodal//BLIP

BLIP (Bootstrapping Language-Image Pre-training) is Salesforce's pair of language-image models, BLIP (2022) and BLIP-2 (2023). The innovation of BLIP-2 is the Q-Former, a lightweight querying transformer that bridges a frozen vision encoder to a frozen LLM.

BLIP (Bootstrapping Language-Image Pre-training) is Salesforce's pair of language-image models, BLIP (2022) and BLIP-2 (2023). The innovation of BLIP-2 is the Q-Former, a lightweight querying transformer that bridges a frozen vision encoder to a frozen LLM.

Neither the vision model nor the language model needs fine-tuning: only the Q-Former trains, which makes it extremely parameter-efficient. It superseded CLIP for many tasks, because CLIP only matches images to text by contrast while BLIP also generates (captions, answers), and SigLIP (Google) and PaliGemma are related successors.