ML//transformer//weight tying

Weight tying is a parameter-sharing technique in which one matrix does two jobs, so that the same table that translates words into vectors also translates vectors back into words, transposed. The embedding matrix \(E\) that converts token IDs into vectors is reused, as \(E^{\mathsf T}\), as the LM head that converts the final vector back into vocabulary scores.

Weight tying is a parameter-sharing technique in which one matrix does two jobs, so that the same table that translates words into vectors also translates vectors back into words, transposed. The embedding matrix EEE that converts token IDs into vectors is reused, as ETE^{\mathsf T}ET, as the LM head that converts the final vector back into vocabulary scores.

It is conceptually elegant: the embedding says what a token means in vector space, and the LM head says which token is closest to a vector in the same space.

It reduces the parameter count significantly, because the embedding matrix is huge (vocabulary size times embedding dimension).

It works because the model learns to place semantically similar tokens near each other in the shared space, and the last vector has been refined through so many layers that it knows exactly what it is relative to the entire vocabulary, which is sufficient to predict the next token.