TabPFN
Forecasting & tabular · Jul 2022
A foundation model for spreadsheet-sized data. You don’t train it on your table: you pass your labelled rows as context and it predicts the new rows in one forward pass, approximating Bayesian inference.
The statistical view
The posterior predictive distribution: the probability of the label of a new row given your labelled rows, averaged over every data-generating process its prior allows.
p(y | x, D) = ∫ p(y | x, φ) · p(φ | D) dφ
- Loss
- Trained across synthetic datasets to minimise −log q_θ(y_test | x_test, D_train). Minimised over datasets drawn from the prior, this makes q_θ approximate the true posterior predictive.
- Answering
- A full distribution per row: class probabilities, or for regression a histogram over bins, so uncertainty comes with every prediction.
What does the number do next?
The same machinery pointed at numbers: series become tokens or patches, and a table of labelled rows becomes context for predicting new ones.
A demonstration of the objective, computed in your browser on illustrative data. It is not TabPFN running here.
How it is trained
- 1Define a prior over datasets: random causal graphs, functions and noise that generate synthetic tables.
- 2Sample a dataset and split it into training and test rows.
- 3Train the Transformer to predict the test labels, given the training rows as context.
- 4Repeat over millions of synthetic datasets, then freeze the weights.
- 5On your data: pass the training rows and new rows through the network. No gradient steps.
Line of descent
- Jul 2022TabPFN — University of Freiburg: classification on up to 1,000 rows, in about a second.
- Jan 2025TabPFN v2 — Published in Nature: regression and classification, up to 10,000 rows and 500 features.
What data teams use it for
- Small and medium tabular problems: churn, risk, clinical studies
- A strong baseline in seconds, with no hyperparameter tuning
- Calibrated probabilities and uncertainty for every prediction
- Missing values and categorical features handled natively
Limits
- Size limits: about 10,000 rows and 500 features for v2
- Every prediction attends to all training rows, so cost grows with the training set
- Only as good as its prior: data unlike the synthetic generator can surprise it
- Needs a GPU for speed near the upper limits
Classic algorithms underneath
The same ideas, small enough to train in a browser.
Same objective
TimesFM
Google Research · Oct 2023
Google’s zero-shot forecaster. It reads a series as a sequence of patches and predicts the next patch, the time-series version of next-token prediction, so it can forecast series it has never seen.
Chronos
Amazon · Mar 2024
Treats forecasting as language modelling. Values are scaled and rounded into a fixed vocabulary of bins, an off-the-shelf language model learns the resulting tokens, and sampling many continuations gives a probabilistic forecast.