9  Word Embeddings: From One-Hot to Distributed Representations

TipTL;DR

A one-hot vector encodes identity but nothing else — “cat” and “dog” are as far apart as “cat” and “galaxy.” A word embedding maps each token to a dense vector in \(\mathbb{R}^d\) where geometry carries meaning: similar words cluster, analogies become vector arithmetic. Hinton proposed distributed representations in 1986; Mikolov’s Word2Vec (2013) made them practical at scale. Every RNN and Transformer in this timeline takes embeddings as input.

Depends on: MLP & Backpropagation

9.1 Why this matters

An RNN processes a sequence token by token. Before the first recurrent step, each token must become a vector the network can do arithmetic on. The naive encoding — a one-hot vector of size \(|V|\) — is both wasteful (sparse, high-dimensional) and uninformative (all tokens are equidistant). Every downstream architecture in this timeline assumes a denser, more expressive input.

9.2 The mechanism

9.2.1 One-hot encoding

For a vocabulary \(V\) with \(|V|\) tokens, a one-hot vector \(\mathbf{e}_i\) places a \(1\) at position \(i\) and \(0\) everywhere else:

\[ \mathbf{e}_i \in \{0, 1\}^{|V|}, \quad (\mathbf{e}_i)_j = \mathbf{1}[j = i]. \]

Multiplying by an embedding matrix \(\mathbf{E} \in \mathbb{R}^{d \times |V|}\) is equivalent to a lookup:

\[ \mathbf{x}_i = \mathbf{E}\,\mathbf{e}_i = \mathbf{E}_{:,i} \in \mathbb{R}^d. \]

In practice the one-hot step is skipped entirely — the embedding layer stores \(\mathbf{E}\) and returns column \(i\) directly.

9.2.2 Distributed representations

Hinton (Hinton et al., 1986) observed that a learned feature representation — where each dimension captures a graded property rather than a single identity — generalises better than local representations. The embedding matrix \(\mathbf{E}\) is learned end-to-end alongside the rest of the network; no hand-crafted features are needed.

9.2.3 Word2Vec: learning from context

Mikolov et al. (Mikolov et al., 2013) showed that a shallow network trained to predict context from a word (or vice versa) learns embeddings with striking geometric structure. Two objectives:

  • Skip-gram: given a center word \(w\), predict surrounding context words \(c\) within a window: \[ \mathcal{L} = -\sum_{(w,c)} \log P(c \mid w), \quad P(c \mid w) = \frac{\exp(\mathbf{v}_c^\top \mathbf{u}_w)}{\sum_{c'} \exp(\mathbf{v}_{c'}^\top \mathbf{u}_w)}. \]
  • CBOW (Continuous Bag of Words): predict the center word from averaged context vectors — faster to train, slightly weaker at rare words.

The learned vectors obey the famous analogy: \[ \mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}. \]

9.2.4 GloVe: global co-occurrence

Pennington et al. (Pennington et al., 2014) replaced the local context window with a global co-occurrence matrix \(X\), fitting embeddings so that: \[ \mathbf{u}_i^\top \mathbf{v}_j + b_i + b_j \approx \log X_{ij}. \]

GloVe and Word2Vec produce comparably useful embeddings; GloVe is faster to train on large corpora.

9.2.5 Minimal sketch

The full operation is a single matrix column lookup. Below: vocabulary of 10 tokens, embedding dimension \(d = 4\), random initialization. Calling embed(i) returns the \(i\)-th column of \(\mathbf{E}\) — the dense vector the network will process instead of a one-hot.

import numpy as np

vocab_size, d = 10, 4
rng = np.random.default_rng(0)
E = rng.normal(scale=0.01, size=(d, vocab_size))

def embed(token_id):
    return E[:, token_id]

print(embed(3))   # dense vector for token id 3
[ 0.001049   -0.00218792  0.0035151   0.00540846]
WarningPitfall: embedding dimensionality

\(d\) is a hyperparameter: too small and the space is too compressed to encode vocabulary structure; too large and the model overfits or trains slowly. Common choices: 128–512 for task-specific models, 768–4096 for large LMs.

9.3 Application & impact

Concept here What it becomes Where
Lookup table \(\mathbf{E}\) Token embedding layer Every RNN, Transformer
Distributed representation Contextualised representation BERT hidden states, GPT activations
Word2Vec skip-gram loss Contrastive / noise-contrastive losses SimCSE, sentence embeddings
Fixed pretrained vectors Fine-tuned embeddings Every modern LM
  • Word2Vec and GloVe were the dominant input representation for RNN-based MT (2014–2016).
  • The Transformer later learned its own embeddings end-to-end, but the lookup-table interface — token id → dense vector — is unchanged.
  • Positional embeddings extend this to encode position alongside identity — covered in Positional Encoding.
NoteKey takeaway

The embedding layer converts discrete tokens into continuous geometry. Everything in the RNN and Transformer eras assumes this step; the only thing that changes downstream is what the network does with those vectors.

→ Next: Vanilla RNN & BPTT