Embedding

An embedding is a learned representation of data as a dense continuous vector. It maps high-dimensional, discrete, or symbolic inputs (words, tokens, users, items, images) into a space where geometry—distance and direction—reflects semantic or structural similarity.

In a language model, the embedding table maps each token id to a vector that enters the Transformer stack. The final LM head maps hidden vectors back to vocabulary logits (sometimes with tied weights). For retrieval, a dedicated embedding model encodes queries and passages for dense retrieval in a vector database.

Why use embeddings

  • Compress discrete symbols into fixed-size vectors
  • Enable similarity search (cosine, dot product)
  • Transfer structure learned on large corpora to downstream tasks

See: word embedding, matrix embedding. Training objectives (masked LM, contrastive pairs, next-token) shape what “nearby” means in the space.

← Back to AI Glossary