Gemini
Next-token prediction · Dec 2023
Google DeepMind’s natively multimodal models. Text, images, audio and video are turned into tokens in one sequence, so a single next-token model reasons across all of them.
The statistical view
The next token of an interleaved sequence in which image patches, audio and video frames have been encoded as tokens alongside text.
p(xₜ | x<ₜ), x = [text, image, audio, video tokens]
- Loss
- Next-token cross-entropy over the mixed sequence, so a question about a chart and the chart itself sit in the same context.
- Answering
- Text tokens sampled one at a time. A router sends each token to a few expert sub-networks, so only part of the model runs per token.
What comes next?
Cover everything after position t and predict it. Repeated over a corpus, this is maximum-likelihood estimation of the distribution of text.
A demonstration of the objective, computed in your browser on illustrative data. It is not Gemini running here.
How it is trained
- 1Encode every modality into a shared token sequence.
- 2Pre-train with next-token prediction across modalities.
- 3Mixture-of-experts: a learned router activates a few expert networks per token, adding capacity without adding compute per token.
- 4Instruction tuning and reinforcement learning from human feedback.
Line of descent
- Dec 2023Gemini 1.0 — Ultra, Pro and Nano sizes.
- Feb 2024Gemini 1.5 — Mixture-of-experts and a context of up to 1 million tokens.
- Feb 2024Gemma — Smaller open-weight models built from the same research.
- Mar 2025Gemini 2.5 — A “thinking” model that reasons before answering.
What data teams use it for
- Asking questions of chart screenshots, dashboards and scanned PDFs
- Very long context: a whole codebase or long logs in one prompt
- Transcribing and analysing audio and video
- Built into BigQuery and Google Workspace
Limits
- Can hallucinate, and reading values off a chart image is approximate
- Parameter count and training mix are not published
- Long contexts are expensive, and recall is uneven across very long inputs
- Only available through Google’s services
Classic algorithms underneath
The same ideas, small enough to train in a browser.
Same objective
GPT
OpenAI · Jun 2018
The model family that made next-token prediction famous: a Transformer trained to guess the next piece of text, then tuned with human feedback to follow instructions.
Llama
Meta · Feb 2023
Meta’s open-weight language models. The same next-token objective as GPT, but the weights can be downloaded, inspected, fine-tuned and run on your own hardware.
Claude
Anthropic · Mar 2023
Anthropic’s family of language models: pre-trained on next-token prediction, then aligned with human feedback and Constitutional AI, where the model critiques answers against a written set of principles.