Claude
Next-token prediction · Mar 2023
Anthropic’s family of language models: pre-trained on next-token prediction, then aligned with human feedback and Constitutional AI, where the model critiques answers against a written set of principles.
The statistical view
Pre-training learns the next-token distribution, like every model in this group. Alignment then learns a second model: how likely one answer is to be preferred over another.
P(A preferred to B) = σ(r(A) − r(B))
- Loss
- Pre-training: L = −Σₜ log p_θ(xₜ | x<ₜ). The preference model fits the Bradley–Terry formula above to ranked pairs of answers, by logistic loss on the reward difference.
- Answering
- Tokens sampled from the tuned distribution. With tools, numbers can be computed by code the model writes rather than generated token by token.
What comes next?
Cover everything after position t and predict it. Repeated over a corpus, this is maximum-likelihood estimation of the distribution of text.
A demonstration of the objective, computed in your browser on illustrative data. It is not Claude running here.
How it is trained
- 1Pre-train on next-token prediction over a large text and code corpus.
- 2Fine-tune on demonstrations of helpful, honest answers.
- 3Train a preference model on pairs of answers ranked by people, or by the model itself judging against a constitution (RLAIF).
- 4Optimise the language model with reinforcement learning against that preference model.
Line of descent
- Dec 2022Constitutional AI — Paper: AI feedback guided by written principles replaces much of the human labelling.
- Mar 2023Claude — First public release.
- Jul 2023Claude 2 — Public chat interface and a 100K-token context window.
- Mar 2024Claude 3 — Haiku, Sonnet and Opus tiers. Image input.
- May 2025Claude 4 — Opus 4 and Sonnet 4, aimed at long coding and agent tasks.
What data teams use it for
- Reading long documents: data contracts, schemas, policies
- Agentic data work: writing and running SQL or Python with tools
- Structured extraction from messy text to JSON
- Reviewing code and pipeline changes
Limits
- Can hallucinate, like any language model
- Parameter count and training mix are not published
- Numbers in an answer are generated, not computed, unless it runs a tool
- At temperature above 0 the same prompt can give different answers
Classic algorithms underneath
The same ideas, small enough to train in a browser.
Same objective
GPT
OpenAI · Jun 2018
The model family that made next-token prediction famous: a Transformer trained to guess the next piece of text, then tuned with human feedback to follow instructions.
Llama
Meta · Feb 2023
Meta’s open-weight language models. The same next-token objective as GPT, but the weights can be downloaded, inspected, fine-tuned and run on your own hardware.
Gemini
Google DeepMind · Dec 2023
Google DeepMind’s natively multimodal models. Text, images, audio and video are turned into tokens in one sequence, so a single next-token model reasons across all of them.