What Is ControlNet?

A diffusion model's image-generating network already has learned weights (see what a diffusion model is), and ControlNet leaves those weights completely locked. Instead, it duplicates the relevant part of that network into a second, trainable copy, and feeds the structural input, a pose skeleton, a Canny edge map, a depth image, into that copy. The trainable copy's output is then injected back into the frozen model at each layer, steering generation without ever updating the original weights.

This locked-plus-trainable split is what makes ControlNet practical: the base model's broad visual knowledge, learned from massive training, stays intact and reusable, while the added control signal only has to learn how to translate one specific kind of structure (a pose, a depth map) into an influence on that frozen network's generation.

A pose-conditioned ControlNet is fed keypoints from a pose estimator, joints and limbs extracted from a reference image or drawn by hand, and generation follows that exact skeleton: same stance, same limb angles, regardless of what the text prompt says about the scene. A depth-conditioned ControlNet is fed a depth map instead, forcing generated content to respect which parts of the scene should be near or far from the camera, useful for keeping a background layout fixed while changing everything about its style.

Both cases share the same underlying trick: the control signal is converted into a feature-space representation matching the base model's internal resolution, then added into the denoising process at every step, the same way a text prompt's embedding steers denoising (see how AI image generation works), just carrying spatial structure instead of semantic content.