ML//multimodal//LLaVA

LLaVA (Large Language and Vision Assistant) is the late-fusion approach to visual language models: a CLIP vision encoder, a linear projection and a LLaMA language model: CLIP extracts the image features and the linear layer projects them into the LLM's embedding space. It is simple, cheap and effective.

LLaVA (Large Language and Vision Assistant) is the late-fusion approach to visual language models: a CLIP vision encoder, a linear projection and a LLaMA language model: CLIP extracts the image features and the linear layer projects them into the LLM's embedding space. It is simple, cheap and effective.

It is trained by visual instruction tuning, a fine-tune on image-text pairs whose text includes instructions. It spawned a large family (LLaVA-1.5, LLaVA-NeXT and countless variants) and was the Alpaca moment for multimodal models.