ML//VAE

A VAE (variational autoencoder) is a generative neural network that learns to compress data into a structured latent space and then generates new data by sampling from that space. An encoder maps the input to a latent distribution, and a decoder maps a latent sample to an output.

A VAE (variational autoencoder) is a generative neural network that learns to compress data into a structured latent space and then generates new data by sampling from that space. An encoder maps the input to a latent distribution, and a decoder maps a latent sample to an output.

What makes it variational is that the encoder outputs a mean and a variance (a Gaussian distribution) instead of a single point. Training samples from this distribution, which forces the latent space to be smooth and continuous.

The KL divergence loss is the key: it penalizes the encoder for making the latent distribution too different from a standard Gaussian. This regularization is what makes the latent space navigable; without it the model is a plain autoencoder (good reconstruction, but holes in the latent space).

Latent diffusion (including Stable Diffusion) uses a pre-trained VAE to compress images from pixel space (e.g. 512×512×3512\times512\times3512×512×3) to latent space (e.g. 64×64×464\times64\times464×64×4). The diffusion model then operates in this much smaller space, making training and generation dramatically cheaper.

Against GANs, VAEs produce blurrier outputs but have stable training and a meaningful latent space. GANs produce sharper outputs but suffer from mode collapse and training instability. Latent diffusion combines the best of both worlds: VAE for compression, diffusion for generation quality.

The decoder half is also used in text-to-image: after the diffusion model denoises in latent space, the VAE decoder converts back to pixel space. This is why all text-to-image models based on latent diffusion share a VAE component.