How Does AI Video Upscaling Work?

At its core, a video upscaler does what an image super-resolution model does: predict plausible high-frequency detail, texture, sharp edges, grain, that a genuine high-resolution version of the footage would have, rather than just interpolating existing pixels. Run naively, frame by frame, independently, each frame gets its own separately generated detail.

Diffusion-based upscalers add randomness at each denoising step by design (see what a diffusion model is), which is exactly what causes the flicker: two adjacent frames showing the same static object can each get subtly different generated texture, because each frame's denoising process drew from independent randomness, with nothing forcing the two results toward agreement.

Upscale-A-Video, one system that established this approach for diffusion-based video super-resolution, combines a local and a global mechanism. Locally, temporal layers are added directly into the network (both the main denoising network and the decoder that turns latents back into pixels), so nearby frames' detail generation is directly linked rather than computed independently, holding short sequences consistent with each other. Globally, a separate technique propagates and blends latent representations across the entire clip, without needing additional training, using optical flow (the same frame-to-frame motion signal behind frame interpolation, see how AI frame interpolation works) to track where a pixel's detail should carry forward to as it moves.

Other approaches use a recurrent design instead: each frame's upscaling is conditioned not just on that frame's own low-resolution input, but also on the network's own output from the previous frame, warped forward to account for motion, giving each new frame's generated detail an explicit anchor to stay consistent with rather than starting fresh.