ML//transformer//attention//projection matrices

The projection matrices are the learned weight matrices that create Q, K, V from the raw embeddings, and the one that recombines the heads afterwards.

The projection matrices are the learned weight matrices that create Q, K, V from the raw embeddings, and the one that recombines the heads afterwards.

WQW_QWQ​Projects the embeddings into query space (what to look for)

WKW_KWK​Projects them into key space (what to advertise)

WVW_VWV​Projects them into value space (what to transmit), learning a selective compression that keeps only the subspace useful for its head

WOW_OWO​The output projection, which recombines the outputs of all heads back to the model dimension

Each head has its own WQW_QWQ​, WKW_KWK​ and WVW_VWV​, initialized randomly and then pushed to specialize by backprop. They project from the model dimension down to the head dimension (in GPT-3, from 12288 to 128 per head). LoRA is typically applied to these four matrices, because attention is where behavior changes most with the fewest parameters.