Llama
Next-token prediction · Feb 2023
Meta’s open-weight language models. The same next-token objective as GPT, but the weights can be downloaded, inspected, fine-tuned and run on your own hardware.
The statistical view
The next-token distribution. With the weights in hand you can read that whole distribution, the log-probability of every candidate token, not just the text that was sampled.
log p(y | x) = Σₜ log p(yₜ | x, y<ₜ)
- Loss
- L = −Σₜ log p_θ(xₜ | x<ₜ) in pre-training. Llama 3 was aligned with direct preference optimisation (DPO), which fits human preferences without a separate reward model.
- Answering
- Sampled text, or scores: the summed log-probability of a label or answer, useful for zero-shot classification and perplexity.
What comes next?
Cover everything after position t and predict it. Repeated over a corpus, this is maximum-likelihood estimation of the distribution of text.
A demonstration of the objective, computed in your browser on illustrative data. It is not Llama running here.
How it is trained
- 1Tokenise text with a byte-pair vocabulary (128K tokens in Llama 3).
- 2Pre-train with next-token prediction on trillions of tokens of public data.
- 3Supervised fine-tuning on instruction and chat examples.
- 4Preference tuning: rejection sampling and PPO for Llama 2, DPO for Llama 3.
Line of descent
- Feb 2023LLaMA — 7B to 65B, for researchers. The 13B model beat GPT-3 on most reported benchmarks.
- Jul 2023Llama 2 — Commercial use allowed. Chat versions tuned with RLHF.
- Apr 2024Llama 3 — 8B and 70B, trained on over 15 trillion tokens.
- Jul 2024Llama 3.1 — 405B, the largest open-weight model at the time.
- Apr 2025Llama 4 — Scout and Maverick: mixture-of-experts, natively multimodal.
What data teams use it for
- Private deployments where data can’t leave your infrastructure
- Fine-tuning a small model for one classification or extraction task (LoRA)
- Scoring text with log-probabilities: perplexity, zero-shot labels
- Embedding and retrieval pipelines
Limits
- You host it: GPUs, latency and scaling are your problem
- The licence has conditions: open weights, not open source
- Training data is described but not released
- Smaller variants trail frontier closed models on hard reasoning
Classic algorithms underneath
The same ideas, small enough to train in a browser.
Same objective
GPT
OpenAI · Jun 2018
The model family that made next-token prediction famous: a Transformer trained to guess the next piece of text, then tuned with human feedback to follow instructions.
Claude
Anthropic · Mar 2023
Anthropic’s family of language models: pre-trained on next-token prediction, then aligned with human feedback and Constitutional AI, where the model critiques answers against a written set of principles.
Gemini
Google DeepMind · Dec 2023
Google DeepMind’s natively multimodal models. Text, images, audio and video are turned into tokens in one sequence, so a single next-token model reasons across all of them.