How Do AI Audio Models Work? TTS, STT, and Speech-to-Speech

Most neural TTS systems split the job into two models. First, an acoustic model reads the input text (broken into phonemes, the distinct sound units of speech) and predicts a mel spectrogram: a 2D representation of the audio with time on one axis and frequency on the other, closer to how the ear perceives sound than a raw waveform is. Models like Tacotron 2 and FastSpeech 2 do this step, and FastSpeech 2 also predicts pitch, duration, and energy for each sound, which is what gives generated speech natural-sounding rhythm instead of a flat monotone.

Second, a vocoder converts that spectrogram into an actual audio waveform, the numbers that get played through a speaker. Early neural vocoders like WaveNet generated audio one sample at a time, which is accurate but slow; later ones like WaveGlow and HiFi-GAN generate a whole chunk of waveform in one pass, trading a little quality for speed that makes real-time speech generation practical.

This is also why a TTS voice can be cloned from a short reference clip: the acoustic model is conditioned on a speaker embedding, a vector that captures voice characteristics, extracted from the reference audio, alongside the text it's converting.

Speech-to-text runs the opposite direction and, in OpenAI's Whisper, uses a fairly standard encoder-decoder transformer, the same family of architecture behind machine translation. Input audio is chopped into 30-second chunks and converted into a log-Mel spectrogram, the same kind of time-frequency representation TTS predicts, just used here as input instead of output.

The encoder passes that spectrogram through convolutional layers, which pick up local patterns like individual phonemes, and then through transformer layers, which model longer-range structure: how sounds combine into words and words into sentences. The decoder then generates the transcript one token at a time, conditioned on what the encoder extracted and on the tokens it has already produced, the same next-token approach behind a text LLM, just conditioned on audio instead of a text prompt.

Because Whisper was trained on hundreds of thousands of hours of multilingual, multitask audio (transcription and translation both), the same model handles many languages and can translate speech in one pass instead of needing a transcribe-then-translate pipeline.

Before native audio models, a voice assistant was three separate systems chained together: speech-to-text turns what you said into a transcript, a text LLM reads that transcript and writes a text reply, and text-to-speech turns the reply back into audio. This is a cascaded pipeline, and it's still how most voice products work.

It has two real costs. Latency stacks up across three sequential models, typically 1.5 to 3 seconds from the end of your sentence to the start of the reply in production systems, which is noticeably slower than a natural conversational pause. And converting speech to text is lossy: a transcript keeps the words but throws away tone, hesitation, emphasis, whether you sounded frustrated or amused, information a text-only LLM never gets to use in deciding how to respond.

Newer models, including OpenAI's Realtime API models and other speech LLMs, process audio input and produce audio output directly, without an explicit transcription step in between. The general mechanism behind this class of model, documented in the research literature, is discrete audio tokenization: a neural audio codec compresses a waveform into a sequence of discrete tokens, the same kind of token a text LLM operates on, just representing chunks of sound instead of chunks of text. A single transformer can then be trained on sequences that mix audio tokens and text tokens, learning to predict the next token regardless of which modality it belongs to, and generate a reply as a token sequence that decodes back into a waveform.

OpenAI has not published the internal architecture of its native audio models, so treat that tokenization mechanism as the general approach current speech-LLM research is built on, not a confirmed description of any specific OpenAI model. What is confirmed is the behavior: audio goes in, a response is generated, audio comes out, with no separate text transcript step exposed in between.

This is also why native audio models can react to tone and pacing a cascaded system can't: nothing about how you said something gets discarded on the way in, because it's never converted to text-only tokens at all.

Native speech-to-speech wins on latency and on preserving paralinguistic information. It does not automatically win on quality: independent evaluations have found cascaded pipelines still outperform end-to-end speech-to-speech models on tasks like following complex instructions or reasoning correctly about what was said, likely because the text-based reasoning step in a cascade is doing real work that a single audio-native model hasn't yet learned to match. Which one is the right architecture depends on whether the product needs low-latency natural conversation or high-accuracy task completion more.