GoogleSample
BERT
Masked reconstruction · Oct 2018
The model that showed reading in both directions pays off. Hide some words, predict them from the context on both sides, and the network learns representations that transfer to almost any language task.
Released
Oct 2018
Access
Open source
Parameters
110M (Base), 340M (Large)
Source
Architecture
Encoder-only, bidirectional Transformer
Training data
BooksCorpus (800M words) and English Wikipedia (2,500M words).
Text
The statistical view
The probability of each hidden token given all the visible tokens, on its left and its right.
p(xᵢ | x∖M) for every masked position i ∈ M
- Loss
- L = −Σ_{i∈M} log p_θ(xᵢ | x∖M), computed only on the 15% of positions that were masked. The original also predicted whether sentence B follows sentence A.
- Answering
- Not a text generator. It outputs a vector per token; a small head on top, often a logistic regression, turns that into a label, a score or a span of text.
What is missing here?
Hide a slice of the input and rebuild it from what remains on both sides. The point is not the guess but the representation learned along the way.
A demonstration of the objective, computed in your browser on illustrative data. It is not BERT running here.
How it is trained
- 1Tokenise into WordPieces and add [CLS] and [SEP] markers.
- 2Pick 15% of tokens: 80% become [MASK], 10% a random token, 10% stay unchanged, so the model can’t rely on seeing [MASK].
- 3Predict the original tokens from both sides of context.
- 4Fine-tune: add one output layer and train the whole network briefly on labelled data.
Line of descent
- Oct 2018BERT — State of the art on 11 language tasks at release.
- Jul 2019RoBERTa — Meta drops next-sentence prediction and trains longer on more data.
- Oct 2019DistilBERT — Hugging Face: 40% smaller, keeps 97% of the language understanding.
- Oct 2019BERT in Google Search — Used to understand search queries.
What data teams use it for
- Text classification: sentiment, topic, intent
- Named-entity recognition: names, products, amounts in free text
- Sentence embeddings for search and deduplication (Sentence-BERT)
- Cheap to run: batch jobs fit on a CPU
Limits
- Context limited to 512 tokens
- Not generative: it fills blanks, it doesn’t write
- Needs labelled data to fine-tune for each task
- English only in the original release; multilingual versions came later
Classic algorithms underneath
The same ideas, small enough to train in a browser.
