What Is a Neural Audio Codec?

A traditional codec like MP3 compresses audio by throwing away detail a human ear is unlikely to notice, using a fixed, hand-designed formula. A neural audio codec does something different: it learns, from data, how to compress a waveform into a short sequence of discrete tokens, then reconstruct audio that sounds close to the original from just those tokens.

Discrete means each token is one of a fixed, finite set of symbols, like a word from a vocabulary, rather than a continuous number. That's the detail that matters most: once sound is represented as discrete tokens, it looks structurally like text to a transformer, a sequence of symbols from a fixed vocabulary, which is exactly the format a language model already knows how to predict.

Models like SoundStream and EnCodec follow the same three-stage shape. An encoder, a stack of convolutional layers, compresses the raw waveform into a shorter sequence of continuous vectors. A quantizer then snaps each vector to the nearest entry in a learned codebook, a fixed list of reference vectors, and what gets kept is just the index of that entry: a discrete token. A decoder, mirroring the encoder, takes the token sequence and reconstructs a waveform.

The whole system is trained end-to-end so the reconstructed audio matches the original as closely as possible, with the codebook itself learned during training rather than designed by hand.

Snapping a vector to one codebook entry loses information, the gap between the vector and its nearest codebook match. Residual vector quantization (RVQ), the technique both SoundStream and EnCodec use, fixes this by chaining several codebooks: the first quantizes the vector, a second codebook then quantizes just the leftover error from the first, a third quantizes what's still left after that, and so on.

Each added codebook token buys a bit more fidelity at a bit more storage cost, so a single trained codec can serve several bitrates by simply using more or fewer of its codebooks, without retraining. This is also the direct reason audio tokens can vary in count per second of sound: more codebooks per timestep means more tokens, higher bitrate, and (up to a point) better reconstructed quality.

Once audio is a token sequence, the exact same next-token training approach behind text LLMs applies: a transformer can be trained to predict the next audio token, generating new sound, or trained on a mixed sequence of audio and text tokens, learning to move between the two modalities in one model. That's the mechanism behind native speech-to-speech systems, and the reason audio generation research increasingly looks like language modeling research with a different vocabulary.

It also means a codec's quality sets a ceiling: a downstream model can only generate audio as good as what its codec's decoder can reconstruct from tokens, so codec fidelity, bitrate, and token rate are active research targets in their own right, not settled infrastructure.