Why does the same sentence sometimes cost more tokens in one language than another?

The tokenizer's vocabulary is built from patterns in its training text, which skews toward whichever languages were most represented. A language underrepresented in that training data gets fewer of its common words and word-pieces their own token, so the same sentence splits into more, smaller tokens, and costs more to process.

Can two different tokenizers produce different token counts for the same text?

Yes. Each model ships with its own tokenizer, trained on its own vocabulary, so token IDs and even the boundaries where text gets split are specific to that tokenizer. A count from one model's tokenizer doesn't transfer to another model's.

LLM · STEP 1

What Is Tokenization?

Tokenization is the first thing that happens to any text an LLM processes: the text is sliced into tokens, chunks that are often smaller than a whole word, and each token is swapped for an integer ID from a fixed vocabulary. A neural network operates on numbers, not letters, so this conversion is the entry point for everything downstream.

One-pager titled "Tokenization": a sentence drawn as a row of chips, one word split into two chips to show word does not equal token, each chip wired down to a numeric ID below it, with a REMEMBER band noting a word can be several tokens, spaces live in the token, and the model reads numbers. — One-pager: a sentence sliced into tokens, each one wired down to its integer ID.

A word is not the unit, a token is

A tokenizer doesn't split text at spaces the way a person might expect. Modern LLMs use subword tokenization (byte-pair encoding, or a close variant), which builds its vocabulary by starting from individual characters and repeatedly merging the most frequent adjacent pairs seen in training text, until it has a fixed-size set of common chunks, whole common words, frequent prefixes and suffixes, and, for rare or unfamiliar words, fragments.

That's why a common word like "the" is usually one token, while a less common or compound word can split into two or three: "tokenization" might become " token" and "ization", two tokens for one word. The leading space is typically part of the token itself, not a separate character, which is a detail that trips people up when they try to count tokens by counting words.

Every token becomes a number

Once text is sliced into token strings, each one is looked up in a fixed vocabulary table and replaced by its integer ID, a number that has no inherent meaning on its own, just a fixed slot in the tokenizer's table. This is the entire output of tokenization: a sequence of integers. The model architecture never touches raw characters again until the very end of generation, when the reverse process turns predicted IDs back into readable text.

This is also the reason a fixed vocabulary size matters for cost and speed: a bigger vocabulary means fewer tokens per sentence (since more whole words get their own slot) but a larger, more expensive lookup and output layer; a smaller vocabulary means cheaper layers but more tokens per sentence, since more words get split into pieces.

FAQ

Why does the same sentence sometimes cost more tokens in one language than another?: The tokenizer's vocabulary is built from patterns in its training text, which skews toward whichever languages were most represented. A language underrepresented in that training data gets fewer of its common words and word-pieces their own token, so the same sentence splits into more, smaller tokens, and costs more to process.
Can two different tokenizers produce different token counts for the same text?: Yes. Each model ships with its own tokenizer, trained on its own vocabulary, so token IDs and even the boundaries where text gets split are specific to that tokenizer. A count from one model's tokenizer doesn't transfer to another model's.

Sources

Sennrich et al., 'Neural Machine Translation of Rare Words with Subword Units' (BPE, 2016)

Last updated 2026-09-16

What Is Tokenization?

A word is not the unit, a token is

Every token becomes a number

FAQ

Sources

Related