How Does AI Audio Source Separation Work?

A mixed track is a single waveform: every instrument's sound wave has already been added together, and that addition can't be undone by simple subtraction, because the individual waveforms overlap in ways that aren't separable with basic arithmetic. Source separation has to recover an estimate of each original source from that one combined signal, which is why it's a genuinely hard, learned problem rather than a straightforward audio-editing operation.

Most separation models don't work on the raw waveform directly. They first convert it to a spectrogram, a 2D image with time on one axis and frequency on the other, where brightness shows how much energy is present at each frequency at each moment. That representation matters because a spectrogram is closer to how sound is structured perceptually, and it turns audio separation into something closer to an image-processing problem, letting models built for images (like a U-Net) apply almost directly.

Given the mix's spectrogram, the model predicts a separate mask for each target source, vocals, drums, bass, other. A mask is the same shape as the spectrogram, with values indicating how much of the mix's energy at that time-frequency point belongs to that particular source. Multiplying the mix's spectrogram by the vocals mask, for instance, keeps mostly the vocal energy and suppresses the rest; converting that masked spectrogram back to a waveform gives the isolated vocal stem. Spleeter, released by Deezer, works this way: a U-Net trained on spectrogram masks for each of its target stems.

Newer models like Demucs work more directly in the waveform domain, or in a hybrid of both spectrogram and waveform representations, using a U-Net-style architecture with convolutional and, in later versions, transformer layers, trained end-to-end to output separated waveforms directly rather than predicting masks. Both approaches are learned entirely from data: the model is trained on many mixed tracks paired with their known individual stems, and learns what a mixed vocal, drum, or bass line tends to look like well enough to reconstruct isolated versions from mixes it has never seen.

Instruments overlapping in the same frequency range at the same time, a bass note and a low vocal note, for example, give the model less distinguishing information to work with, which is where separation artifacts come from: bleed (a trace of one source still audible in another's stem) and a watery or phasey texture in the reconstructed audio. Quality is typically measured with source-to-distortion ratio (SDR), a signal-processing metric that scores how close a separated stem is to the true isolated source, and newer models are evaluated by how much they improve that score over Spleeter- and original-Demucs-era baselines.