How Does AI Video Generation Work?

An image diffusion model denoises a single frame. A naive way to make video would be to generate frames one at a time and stitch them together, but nothing then forces frame 40 to agree with frame 39: an object's color, a character's face, the exact camera angle can drift or flicker frame to frame, because each frame was generated as if the others didn't exist.

Video diffusion models solve this by denoising an entire clip as one unit. The model sees noisy versions of every frame in the clip together and predicts the noise across all of them jointly, so information about one frame's content directly shapes what nearby frames look like as they get denoised. Consistency isn't patched on afterward, it comes from the model having the whole clip in view at every denoising step.

OpenAI's Sora, described in its technical report, compresses a video into a latent representation the same way latent image diffusion does, then breaks that compressed video into small 3D chunks: a patch of space and a slice of time together, called a spacetime patch. Those patches become the tokens a transformer operates on, the same way words become tokens for a language model.

This patch-based approach is what lets one model train on and generate videos of different lengths, resolutions, and aspect ratios: a longer or higher-resolution video is just more patches, not a different architecture. The model is a diffusion transformer (DiT): the transformer's attention mechanism lets every patch attend to every other patch across both space and time, which is the mechanism that keeps an object's appearance and the camera's motion coherent across the whole clip.

Until 2025, video and audio were generated separately: a silent clip came out of the video model, then a different tool added music, sound effects, or dialogue in a second pass, with no guarantee the two lined up. Google's Veo 3, released in May 2025, was among the first widely available models to generate audio and video together in a single generation pass rather than as a sequential pipeline.

The research direction behind this (see the Ovi paper on joint audio-video generation, cited below) extends the same spacetime-patch idea to a second, parallel stream of audio tokens: a video-generating transformer and an audio-generating transformer share attention across both streams during denoising, so a footstep, a door closing, or a line of dialogue gets tied to the exact frame it should occur on as both streams are generated together. Google has not published the internal architecture of Veo's audio generation, so treat the joint-attention mechanism as the general research approach current models are built on, not a confirmed description of Veo specifically.

This joint approach is also why lip sync and timed sound effects hold up better in newer models than in a bolt-on audio pass: the audio was never independent of the video to begin with, both came from the same denoising process conditioned on the same prompt.

Video models condition on a prompt the same way image models do (see classifier-free guidance for the mechanism), but open-source video systems like Alibaba's Wan2.1 lean heavily on a second input: a negative prompt describing what to avoid. Instead of comparing the prompted prediction against an empty, unconditioned one, the model compares it against a prediction conditioned on the negative prompt, and pushes generation away from that direction specifically.

In practice this means a long negative prompt ("static, blurry, extra fingers, deformed limbs, walking backwards") does real work in a video pipeline: without it, clips are more prone to exactly the defects it names, since nothing is actively steering the denoising process away from those specific, well-represented failure modes.

Long-range consistency is the hardest unsolved part. A character can still change appearance if they leave and re-enter frame, and physics can drift over a longer clip, a ball that should fall at a constant rate can subtly speed up or teleport, because the model is predicting plausible motion from patterns in training data, not simulating physics. OpenAI's own write-up on Sora describes it as a step toward a general-purpose simulator of the physical world while explicitly noting it does not yet accurately model many basic physical interactions.

Generating audio and video together is also far more compute-intensive than generating video alone, one more reason most current video tools still cap clip length at several seconds to a couple of minutes rather than offering unlimited generation.