Where open tokenizers break Tamil

A Tamil letter is not one character to a computer, and many tokenizers forget that.

Tamil is written in an abugida: most letters are a consonant followed by a vowel sign or a pulli (்), which Unicode stores as separate code points. Recent work showed that pre-tokenizers matching letters with \p{L}+ split such scripts at every mark. We measured what that, and two related failures, cost Tamil in 14 widely used open tokenizers, on 3,000 held-out passages.

Qwen3's pre-tokenizer cuts 98.8% of Tamil words, and it needs 9.99 tokens per word: 4.1× Gemma 3 and 6.0× a Tamil-trained tokenizer. Its successor Qwen3.5 fixed the pattern and grew its vocabulary, and needs 4.28, still 1.75× Gemma 3. A fixed pattern isn't enough on its own: DeepSeek-V3 makes no pattern cuts, yet 21.8% of Tamil words contain a token that ends mid-character.

Three letters, five characters, fifteen bytes.

A reader sees தமிழ் as three letters. Unicode stores five code points, because the vowel sign ி and the pulli ் are combining marks (category M), not letters (category L).

A tokenizer that groups "letters" with \p{L}+ stops at every mark, and cuts a consonant away from its own vowel. Every later step, including byte-pair encoding, can only work inside those pieces.

See the cut on your own Tamil.

This runs Qwen3's word pattern and the fixed pattern in your browser. Type or paste any Tamil.

D4: a tokenizer track that names the kind of damage.

3.1 Five measurements per tokenizer

Tokens per word and bytes per token, read next to vocabulary size. Letter cut: a token boundary just before a vowel sign or pulli, which is not always wrong, because Tamil suffixes often start inside a letter (மரத்தை = மரத்த் + ஐ). Pre-tokenizer cut: the same, caused by the pre-tokenizer alone, always a defect. Byte cut: a token that ends inside a character, always a defect.

Everything is reproducible with pip install "uraikal[tokenizers]" and uraikal tokenizers. Tokenizers are loaded without running any repository code.

A sixfold spread, and one outlier.

4.1 All numbers

Cut rates are shares of Tamil words. IndicBERT v2 fails the round trip only because its decoder keeps WordPiece markers. Llama 3.1 and Aya Expanse are gated and not yet included.

The pattern decides what can merge. The vocabulary decides what did.

5.1 One regular expression explains the outlier

Qwen3 matches words with [^\r\n\p{L}\p{N}]?\p{L}+. A vowel sign isn't in \p{L}, so it ends the run and attaches to the next piece: தமிழ் becomes தம | ிழ | ். The gpt-oss (o200k), Mistral Nemo, DeepSeek-V3 and Qwen3.5 patterns include \p{M}, and make almost no such cuts. This mechanism was reported before us; see prior work.

5.2 Qwen3 to Qwen3.5

Qwen3.5 switched to [\p{L}\p{M}]+ and grew its vocabulary from 152k to 248k entries. Tamil tokens per word fell from 9.99 to 4.28, and byte cuts from 99.5% to 3.2%. Both changes landed together, so this can't say how much each contributed. Qwen3.5 still needs 1.75× Gemma 3's tokens.

5.3 Fixing the pattern is necessary, not sufficient

DeepSeek-V3 keeps words whole yet still ends a token mid-character in 21.8% of Tamil words: its vocabulary has too few Tamil merges to reach character boundaries. mT5 (13.3%) and XLM-R (4.6%) show the same effect through byte fallback.

5.4 Letter cuts aren't the right target

Even efficient tokenizers put a boundary inside a letter in 9–42% of Tamil words, often exactly where a suffix begins. We report pre-tokenizer and byte cuts as defects and letter cuts as a diagnostic.

5.5 What it costs

At the same price per token, a Tamil request costs about 4.1× more on Qwen3 than on Gemma 3, and a context window holds about a quarter as much Tamil.

Who found the mechanism first.

We don't claim the \p{L}+ mechanism. It was reported for OpenAI's cl100k pattern in a 2024 tiktoken issue, shown for Tamil, Sinhala and Hindi by Velayuthan & Sarveswaran (COLING 2025), and characterised across 26 languages by Regmi et al. (2026). What's new here is the Tamil measurement of current tokenizers with the three failures separated, the Qwen3 to Qwen3.5 comparison, and the mid-character rate in tokenizers whose pattern is already fixed.

@misc{gopalan2026tamiltokenizers,
  title  = {Where Open Tokenizers Break Tamil: Pre-tokenizer Cuts, Missing Merges and Mid-Character Tokens},
  author = {Gopalan, Jayakumar},
  year   = {2026},
  note   = {Draft. Zhagaram Labs},
  url    = {https://zhagaramlabs.com/papers/tamil-tokenizers/}
}

Copy BibTeX