OpenAISample
CLIP
Contrastive learning · Jan 2021
Learns one space for pictures and words by matching 400 million images to their captions. Because labels become sentences, it can classify images into categories it was never trained on.
Released
Jan 2021
Access
Open weights
Parameters
Several sizes, image encoders from ResNet-50 to ViT-L/14
Source
Architecture
Two encoders, a Vision Transformer or ResNet for images and a Transformer for text, projected into one shared space
Training data
400 million image–text pairs collected from the internet.
ImageText
The statistical view
Which caption belongs to which image in a batch: a softmax over cosine similarities, in both directions.
p(textⱼ | imageᵢ) = exp(sᵢⱼ / τ) / Σₖ exp(sᵢₖ / τ)
- Loss
- Symmetric cross-entropy (InfoNCE): the average of the image→text and text→image losses, where sᵢⱼ is the cosine similarity and the temperature τ is learned.
- Answering
- Embeddings. For zero-shot classification, embed “a photo of a {label}” for each label and pick the one closest to the image.
Which of these belong together?
Pull pairs that truly go together closer, push the rest apart. No labels needed beyond the pairing itself.
A demonstration of the objective, computed in your browser on illustrative data. It is not CLIP running here.
How it is trained
- 1Batch N image–caption pairs (32,768 per batch in the paper).
- 2Embed images and captions, normalise, and compute the N × N matrix of cosine similarities.
- 3Push the N true pairs on the diagonal up and the N² − N wrong pairings down.
- 4At inference, compare an image against captions you write.
Line of descent
- Jan 2021CLIP — Zero-shot ImageNet accuracy matching a supervised ResNet-50.
- Apr 2022DALL·E 2 — Generates images from CLIP embeddings.
- Aug 2022Stable Diffusion — Uses CLIP’s text encoder to condition generation.
- Dec 2022OpenCLIP — Open reproductions trained on the public LAION dataset.
What data teams use it for
- Tagging and searching image libraries with a text query
- Zero-shot classification when you have no labelled images
- Finding near-duplicate or off-topic images in a dataset
- Image embeddings as features for downstream models
Limits
- Inherits the biases of uncurated web captions
- Weak at counting, reading exact text and fine spatial relations
- Zero-shot accuracy depends on how the label prompt is worded
- The training data was never released
Classic algorithms underneath
The same ideas, small enough to train in a browser.
