ML//neural network//embedding//directionality

Directionality is the property of an embedding space by which semantic relationships are encoded as directions, and it is what makes search by the angle between vectors work. The gender direction gives \(\text{king} - \text{man} + \text{woman} \approx \text{queen}\), and going from *cat* to *cats* follows the plural direction.

Directionality is the property of an embedding space by which semantic relationships are encoded as directions, and it is what makes search by the angle between vectors work. The gender direction gives king−man+woman≈queen\text{king} - \text{man} + \text{woman} \approx \text{queen}king−man+woman≈queen, and going from cat to cats follows the plural direction.

A dot product of zero means the vectors are perpendicular, which means unrelated; a high dot product means they are aligned and semantically connected.

These directions emerge during training and are never designed by hand: the model discovers its own feature geometry.

The 12K+ dimensions of a typical embedding encode thousands of overlapping directional relationships simultaneously.

This is why cosine similarity works for search: similar concepts point in similar directions.