ML//neural network//embedding

An embedding is the dense vector a model assigns to a discrete token, and it is the form in which a neural network receives symbols it can compute with: the entry point that projects discrete symbols into the continuous latent space where the model works. The token *king*, for instance, becomes a list such as 0.2, -0.5, 0.8 and so on. In large models the typical dimension is 12K and more.

An embedding is the dense vector a model assigns to a discrete token, and it is the form in which a neural network receives symbols it can compute with: the entry point that projects discrete symbols into the continuous latent space where the model works. The token king, for instance, becomes a list such as 0.2, -0.5, 0.8 and so on. In large models the typical dimension is 12K and more.

An embedding is dense where a one-hot encoding is sparse: the one-hot vector gives each token its own axis and no notion of closeness, the embedding packs meaning into a few hundred to a few thousand shared coordinates, so related tokens sit near each other (cosine similarity).

Static and contextual embeddings differ in when the vector is fixed. Word2Vec and GloVe keep one vector per word whatever the sentence; in a transformer the base embedding is only the starting point, and every attention layer rewrites it with its context, so bank by a river and bank with a loan part ways after a few layers.

Word2Vec showed that embeddings capture meaning: king−man+woman≈queen\text{king} - \text{man} + \text{woman} \approx \text{queen}king−man+woman≈queen. A dot product of 0 means the vectors are perpendicular, and perpendicular means unrelated. Directionality encodes gender, plurality and semantics.

Embeddings are learned during training and never crafted by hand: the model discovers its own feature space.

Each forward pass starts from the base embeddings. The enriched representations of the previous pass are not saved (saving them would make it an RNN).

The embedding matrix that produces these vectors is also used, transposed, as the output layer through weight tying.

To look at an embedding space, people flatten it to two dimensions with t-SNE or PCA; the picture is drawn from the vectors afterwards and plays no part in making them.