What Is CLIP?

Before CLIP, computer vision models classified images into a fixed list of categories decided ahead of time, ImageNet's 1,000 classes being the standard example. Language models worked in a completely different space: continuous vectors representing open-ended text. There was no shared coordinate system connecting the two, no way to directly ask a vision model how well an arbitrary sentence describes an arbitrary image.

CLIP, published by OpenAI in 2021, closed that gap by training an image encoder and a text encoder at the same time, on the same objective, so their outputs land in one shared space instead of two separate ones.

CLIP trained on 400 million (image, caption) pairs collected from the internet. Each training batch runs every image through the image encoder and every caption through the text encoder, producing one vector per image and one per caption. The model then compares every image vector against every caption vector in the batch using cosine similarity, a measure of how closely two vectors point in the same direction.

The training objective pulls a matching image and caption's vectors closer together and pushes every non-matching pair apart. Across millions of pairs, the only way to satisfy that objective consistently is for the encoders to represent meaning: an image of a dog and the word "dog" end up near each other not because anyone labeled the concept, but because that's the one arrangement that keeps matching pairs close and everything else far apart.

Once images and text live in the same space, comparing them is just a distance calculation. Zero-shot classification: score an image against a list of candidate captions like "a photo of a cat" and "a photo of a dog", the highest-scoring caption is the model's answer, with no task-specific training needed. This is the same mechanism a search feature uses to find images matching a text query.

The space also supports vector arithmetic on concepts. Subtracting the vector for an image of a man without a hat from the vector for the same man with a hat lands close to the vector for the word "hat" itself, meaning the encoders capture the concept as a consistent direction, not just a label attached to individual examples.

CLIP itself doesn't generate anything. It only encodes: image or text in, a vector out. What it gives a diffusion model is a way to represent a prompt as a point in a space where nearby points mean similar things, which is exactly the signal a denoising step needs to move an image toward what the prompt describes (see how AI image generation works for the conditioning step itself).

DALL-E 2 made this connection explicit: instead of conditioning directly on a text embedding, it trains a second model to invert CLIP's image encoder, going from a CLIP text embedding to a CLIP image embedding, then decodes that into a picture with a diffusion model. Most other text-to-image systems use a CLIP-style text encoder more directly as the conditioning signal.