ML//transformer//tokenizer//embedding matrix

The embedding matrix is a trainable lookup table that maps each discrete token ID to a dense continuous vector (token 4821 becomes a vector of 512 dimensions), and it is where a language model turns the tokenizer's output into something it can compute with. It has no neural network inside: its entries are the trainable parameters themselves.

The embedding matrix is a trainable lookup table that maps each discrete token ID to a dense continuous vector (token 4821 becomes a vector of 512 dimensions), and it is where a language model turns the tokenizer's output into something it can compute with. It has no neural network inside: its entries are the trainable parameters themselves.

The same matrix, transposed, converts the last layer's output back into a similarity with each token in the vocabulary. This is weight tying, and the transposed matrix is the LM head: the same parameters encode and decode.

Its size is vocab_size × embedding_dim. However big the model, the tokenizer (the embedding dimension) bottlenecks comprehension.

The embeddings it produces are the raw starting point: every forward pass resets from these base vectors.