What Are Embeddings?

Embeddings are representations of values or objects such as text, images, audio, designed to be consumed by machine learning models or semantic search algorithms. Essentially, these embeddings are translated into mathematical forms (vectors) according to the factors and categories the objects belong to.

This matters because it lets ML models find objects that are semantically similar: a model finding similar photos or documents based on one it was fed is able to identify relationships between words and other objects using embeddings.

Vectors

Vectors are a list of numbers, where each number indicates where the object lies along a specified dimension. In ML, vectors are used to search for similar objects — usually a vector search algorithm does this by looking for vectors in a vector database that are close to each other.

• Direction of vector: indicates its blend of characteristics. Two vectors pointing in the same direction share a similar ratio of attributes.
• Magnitude (length) of vector: often reflects confidence, frequency, or popularity. A popular show might have a larger vector magnitude pointing in the same direction as a less popular show with similar themes.

How similarity is calculated

• Cosine similarity — measures direction
• Euclidean distance — measures distance between vector endpoints

A useful trick: L2 normalization pulls vectors to the same point on a unit sphere, which mitigates the issue where two vectors point in the same direction but sit very far apart.

Adding more dimensions

To be more specific and add more traits to an object, you add more dimensions. Say a model wants to determine which TV shows are similar and likely to be watched by the same people — episode count, episode duration, and genre classification can all become dimensions, with each show represented as a point along them.

For LLMs, the context of every word becomes an embedding, which is what allows entire sentences and paragraphs to be searched and analysed.