What Is OCR (Optical Character Recognition)?

Traditional OCR engines, Tesseract is the long-running open-source example, work in stages. First, find where text sits on the page (detection). Then split each line into individual characters or short segments (segmentation). Then classify each segment against known letterforms (recognition), originally using template matching and later a trained classifier or a recurrent network reading across a line.

This pipeline is fast and works well on clean, typed text in a simple layout, a printed page, a single-column PDF. It gets worse fast outside that case: handwriting, unusual fonts, skewed or low-quality scans, and complex layouts (multi-column pages, tables, text overlapping images) all break the assumption that characters segment cleanly, since each stage's errors compound into the next.

Newer OCR models remove the separate detection and segmentation steps and treat the whole problem as one sequence-generation task, the same shape as how a diffusion model treats denoising as one learned process instead of hand-coded stages. A vision transformer encodes the input image into a sequence of patch embeddings, the same patch-based idea image and video diffusion models use. A text transformer decoder then generates the output text token by token, conditioned on those embeddings, attending to whichever parts of the image matter for the next token, the exact mechanism a language model uses to attend to earlier words.

TrOCR, from Microsoft Research, is a direct example of this shape: a pretrained image transformer paired with a pretrained text transformer, trained end to end on images paired with their correct transcription, with no explicit character-segmentation step anywhere in the model. Donut goes further and skips text detection entirely, mapping a raw document image straight to structured output.

The newest systems don't stop at a flat transcript. Trained on document images paired with structured targets, Markdown, JSON for a table, LaTeX for an equation, they output the page's structure directly: which text is a heading, which numbers belong to which row of a table, where a caption belongs to which figure. That's a materially different task from character recognition: it requires the model to have learned reading order and layout semantics, not just what each glyph is.

This is also why these models tend to generalize better across languages and layouts than the classic pipeline did. A shared vision-transformer encoder over patches doesn't require redesigning a segmentation step for each new script or layout convention, the model just needs enough training examples covering that case.