GoogleSample
Word2Vec
Contrastive learning · Jan 2013
The ancestor of modern embeddings. It learns one vector per word so that words used in similar contexts end up close together, famously making king − man + woman ≈ queen.
Released
Jan 2013
Access
Open source
Parameters
Two matrices of vocabulary × dimensions, typically 100 to 300 dimensions
Source
Architecture
Shallow network: an embedding layer and a dot product, no hidden layers
Training data
The published vectors: about 100 billion words of Google News, for 3 million words and phrases.
Text
The statistical view
The probability that a context word appears near a centre word, as a function of the dot product of their vectors.
p(c | w) = exp(u_c · v_w) / Σ_c′ exp(u_c′ · v_w)
- Loss
- Negative sampling contrasts each true pair with k random “noise” words: L = −log σ(u_c · v_w) − Σₖ log σ(−uₖ · v_w). This implicitly factorises a shifted pointwise mutual information (PMI) matrix of word co-occurrences.
- Answering
- Just the vectors. Similarity is the cosine between two of them; analogies are vector arithmetic.
Which of these belong together?
Pull pairs that truly go together closer, push the rest apart. No labels needed beyond the pairing itself.
A demonstration of the objective, computed in your browser on illustrative data. It is not Word2Vec running here.
How it is trained
- 1Slide a window over text; each word and its neighbours form (word, context) pairs.
- 2Skip-gram predicts the context from the word; CBOW predicts the word from its context.
- 3Negative sampling: instead of a softmax over the whole vocabulary, contrast each true pair with a few random words.
- 4Keep the learned embedding matrix and discard the rest.
Line of descent
- Jan 2013Word2Vec — Skip-gram and CBOW (Mikolov et al.).
- Oct 2013Negative sampling — Faster training, and vectors for phrases such as “New York”.
- Oct 2014GloVe — Stanford: factorises co-occurrence counts directly.
- Dec 2014The PMI link — Levy and Goldberg show skip-gram with negative sampling factorises shifted PMI.
What data teams use it for
- Similarity features for categories with many levels: products, job titles
- Embedding baskets or sessions by treating items as words (item2vec)
- Cheap baselines for text classification
- Understanding embeddings: the same idea runs through every model here
Limits
- One vector per word: “bank” (river) and “bank” (money) share one
- Words not seen in training get no vector
- Encodes the biases of its corpus, such as gendered job analogies
- Superseded by contextual embeddings for most text tasks
Classic algorithms underneath
The same ideas, small enough to train in a browser.
