What Is Segment Anything (SAM)?

SAM splits the job into three components. An image encoder, a large vision transformer, processes the input image once into a rich feature representation, an expensive step done a single time regardless of how many objects get segmented afterward. A separate, lightweight prompt encoder turns a click, a box, or text into its own representation. A mask decoder then combines the image features and the prompt representation to output a segmentation mask, the exact set of pixels belonging to the prompted object, in a fast, lightweight final step.

Splitting the expensive image encoding from the cheap prompt-to-mask step is what makes SAM interactive: after the one-time cost of encoding an image, a user can click different objects and get a new mask back in a fraction of a second each time, since only the fast decoder step reruns.

SAM's zero-shot ability, segmenting object categories it was never explicitly trained on, comes from the scale and construction of its training data. The SA-1B dataset behind it holds over a billion segmentation masks across 11 million images, built in three stages: human annotators clicked and refined masks by hand first, then annotators focused on cases SAM couldn't confidently segment yet to broaden the model's coverage, and finally, once the model was good enough, a large majority of masks were generated automatically by prompting SAM itself with a grid of points across each image and keeping the confident results.

That progression, human-labeled, then human-assisted, then mostly model-generated, is what let the dataset reach a scale no fully manual annotation effort could match, and that scale is what gives the model its broad, category-agnostic segmentation ability.

A single point click can be genuinely ambiguous: a click on a person's shirt could mean "segment the shirt," "segment the person," or "segment the person and everything they're holding." Rather than forcing one interpretation, SAM is trained to output multiple candidate masks at different plausible scales for an ambiguous prompt, along with a confidence score for each, so the ambiguity is surfaced rather than silently resolved by an arbitrary default.