# Where open tokenizers break Tamil

Pre-tokenizer cuts, missing merges and mid-character tokens. A paper by Jayakumar Gopalan, Zhagaram Labs. Draft, not yet on arXiv.

- Paper (PDF): https://zhagaramlabs.com/papers/tamil-tokenizers/paper.pdf
- Code: https://github.com/zhagaramlabs/uraikal (`pip install "uraikal[tokenizers]"`, then `uraikal tokenizers`)

## Abstract

Tamil is written in an abugida: most letters are a consonant followed by a vowel sign or a pulli (்), which Unicode stores as separate code points. Recent work showed that pre-tokenizers matching letters with `\p{L}+` split such scripts at every mark. We measured what that, and two related failures, cost Tamil in 14 widely used open tokenizers, on 3,000 held-out passages.

Qwen3's pre-tokenizer cuts 98.8% of Tamil words, and it needs 9.99 tokens per word: 4.1× Gemma 3 and 6.0× a Tamil-trained tokenizer. Its successor Qwen3.5 fixed the pattern and grew its vocabulary, and needs 4.28, still 1.75× Gemma 3. A fixed pattern isn't enough on its own: DeepSeek-V3 makes no pattern cuts, yet 21.8% of Tamil words contain a token that ends mid-character.

## Method

- Held-out web Tamil: 2,000 passages, 113,875 words, from shard `data-52` of Sangraha's verified Tamil split (CC BY 4.0), kept out of all Zhagaram tokenizer training.
- Literature: 1,000 passages, 48,044 words, from Project Madurai. Scores are published, the text isn't.
- Measurements: tokens per word, bytes per token, letter cut (a token boundary before a vowel sign or pulli; a diagnostic), pre-tokenizer cut (the same, caused by the pre-tokenizer; always a defect), byte cut (a token that ends inside a character; always a defect).

## Findings

1. Qwen3 matches words with `[^\r\n\p{L}\p{N}]?\p{L}+`. A vowel sign isn't in `\p{L}`, so தமிழ் becomes தம | ிழ | ். The gpt-oss (o200k), Mistral Nemo, DeepSeek-V3 and Qwen3.5 patterns include `\p{M}` and make almost no such cuts.
2. Qwen3.5 switched to `[\p{L}\p{M}]+` and grew its vocabulary from 152k to 248k. Tamil tokens per word fell from 9.99 to 4.28, and byte cuts from 99.5% to 3.2%. Both changes landed together, so their effects can't be separated yet.
3. Fixing the pattern is necessary, not sufficient: DeepSeek-V3 still ends a token mid-character in 21.8% of Tamil words. mT5 (13.3%) and XLM-R (4.6%) show the same effect through byte fallback.
4. Even efficient tokenizers put a boundary inside a letter in 9–42% of Tamil words, often where a suffix begins, so letter cuts are a diagnostic, not a defect.
5. At the same price per token, Tamil costs about 4.1× more on Qwen3 than on Gemma 3.

## Prior work

The `\p{L}+` mechanism isn't ours. It was reported for OpenAI's cl100k pattern in a 2024 tiktoken issue (https://github.com/openai/tiktoken/issues/292), shown for Tamil, Sinhala and Hindi by Velayuthan & Sarveswaran (COLING 2025, https://aclanthology.org/2025.coling-main.400/), and characterised across 26 languages by Regmi et al. (2026, https://arxiv.org/abs/2608.26449). New here: the Tamil measurement of current tokenizers with the three failures separated, the Qwen3 to Qwen3.5 comparison, and the mid-character rate in tokenizers whose pattern is already fixed.

## Citation

```bibtex
@misc{gopalan2026tamiltokenizers,
  title  = {Where Open Tokenizers Break Tamil: Pre-tokenizer Cuts, Missing Merges and Mid-Character Tokens},
  author = {Gopalan, Jayakumar},
  year   = {2026},
  note   = {Draft. Zhagaram Labs},
  url    = {https://zhagaramlabs.com/papers/tamil-tokenizers/}
}
```