GPT
Next-token prediction · Jun 2018
The model family that made next-token prediction famous: a Transformer trained to guess the next piece of text, then tuned with human feedback to follow instructions.
The statistical view
The probability of the next token given every token before it. Chained together, these conditionals give the probability of a whole text.
p(x₁ … x_T) = ∏ₜ p(xₜ | x₁ … xₜ₋₁)
- Loss
- L = −Σₜ log p_θ(xₜ | x<ₜ): cross-entropy, which makes pre-training maximum-likelihood estimation of the text distribution.
- Answering
- Sample a token from the predicted distribution, append it, repeat. Temperature T reshapes the distribution first: pᵢ ∝ exp(zᵢ / T).
What comes next?
Cover everything after position t and predict it. Repeated over a corpus, this is maximum-likelihood estimation of the distribution of text.
A demonstration of the objective, computed in your browser on illustrative data. It is not GPT running here.
How it is trained
- 1Split text into tokens (sub-word pieces) with byte-pair encoding.
- 2Pre-train: predict each next token over a huge corpus, minimising cross-entropy.
- 3Supervised fine-tuning on example conversations written by people.
- 4Reinforcement learning from human feedback (RLHF): a reward model learned from human rankings of answers steers the model towards preferred responses.
Line of descent
- Jun 2018GPT-1 — 117M parameters. Generative pre-training, then fine-tuning per task.
- Feb 2019GPT-2 — 1.5B parameters, released in stages over misuse concerns.
- May 2020GPT-3 — 175B parameters. Learns new tasks from a few examples in the prompt.
- Nov 2022ChatGPT — A GPT-3.5 model tuned with RLHF for conversation.
- Mar 2023GPT-4 — Accepts images as input. Size and training data no longer published.
- Aug 2025GPT-5 and gpt-oss — A new flagship, and OpenAI’s first open-weight models since GPT-2.
What data teams use it for
- Text-to-SQL and code generation for analysis
- Classifying and extracting fields from free text: tickets, reviews, contracts
- Summarising reports and explaining query results
- Generating synthetic text for testing pipelines
Limits
- Fluent is not the same as correct: it can state false facts with confidence
- The GPT-4 report found the pre-trained model well calibrated, and post-training made its confidence less reliable
- Arithmetic and exact aggregation over large tables are unreliable without tools
- Closed weights and undisclosed training data limit auditing
Classic algorithms underneath
The same ideas, small enough to train in a browser.
Same objective
Llama
Meta · Feb 2023
Meta’s open-weight language models. The same next-token objective as GPT, but the weights can be downloaded, inspected, fine-tuned and run on your own hardware.
Claude
Anthropic · Mar 2023
Anthropic’s family of language models: pre-trained on next-token prediction, then aligned with human feedback and Constitutional AI, where the model critiques answers against a written set of principles.
Gemini
Google DeepMind · Dec 2023
Google DeepMind’s natively multimodal models. Text, images, audio and video are turned into tokens in one sequence, so a single next-token model reasons across all of them.