How Does AI Frame Interpolation Work?

The task splits into two parts. First, optical flow estimation: for every point in the frame, figure out where that point moved to in the next frame, in essence, a motion map showing direction and speed across the whole image. Second, frame synthesis: use that motion map to generate a plausible new frame showing the scene at some fraction of the way between the two, most often exactly halfway.

Older methods computed optical flow from frame A to frame B and from B to A separately, then linearly blended the two to guess the flow at the halfway point. That blend is an approximation, and it tends to produce visible artifacts right at motion boundaries, the edge of a moving hand or a car, where the true in-between motion isn't well described by a straight-line average of the two directions.

RIFE (Real-Time Intermediate Flow Estimation), one of the models that set the current standard for this task, skips the blend-two-flows approach. A neural network (its authors call it IFNet) is trained to predict the intermediate flow directly from the two input frames, the motion at the halfway point itself, rather than inferring it after the fact from two separately computed flows. A second network then uses that predicted flow to actually generate the new frame's pixels.

Because IFNet predicts flow for an arbitrary point in time between the two frames, not just the midpoint, the same model supports generating a frame at any fraction of the way between two originals, useful for smoothing footage to a target frame rate other than a clean doubling.

A ball moving in a straight line, or a camera panning smoothly, gives the flow estimator a clear, consistent motion signal to work with, and the generated in-between frame usually looks correct. Motion that involves occlusion, one object passing in front of another between the two frames, breaks the core assumption: the in-between content behind the moving object was never visible in either original frame, so the model has to invent plausible content for a region it has no real information about.

This is also why interpolation struggles with fast, erratic motion, fast enough that an object's position changes dramatically between frames leaves the flow estimator with a much harder correspondence problem, and small errors in that estimate turn into visible warping or ghosting in the generated frame.