# Leaderboard

Every score here comes from [Uraikal](https://github.com/zhagaramlabs/uraikal), our open Tamil benchmark. We ran each one ourselves with the same public code, so you can check any number, or score your own system the same way.

Each table says whether higher or lower is better. Uraikal is at version 0.1, so treat these numbers as preliminary.

## Speech

### 1. Speech recognition on read speech

**What it measures.** Lower is better. The share of words a system writes wrong, with the share of characters wrong in brackets. Every system is scored after the same Tamil-aware cleanup of the text. FLEURS and Kathbath are public test sets of people reading sentences aloud. Kathbath's new speakers are people missing from its training data.

| System | Type | FLEURS | Kathbath, known speakers | Kathbath, new speakers |
|---|:---|---:|---:|---:|
| [Indic-Transcribe core](https://huggingface.co/bodhan-ai/indic-transcribe-core) | Open weights | 26.1% (12.1%) | 21.5% (3.5%) | 23.4% (4.1%) |
| [IndicConformer 600M](https://huggingface.co/ai4bharat/indic-conformer-600m-multilingual) | Open weights | 27.1% (11.1%) | 21.5% (3.4%) | 23.4% (4.0%) |
| [Whisper large-v3 turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | Open weights | 67.5% (18.5%) | 67.5% (15.9%) | 67.2% (16.0%) |
| [MiniCPM-o 4.5](https://huggingface.co/openbmb/MiniCPM-o-4_5) | Open weights | 140.0% (103.1%) | not run | not run |

**How to read it.** One wrong letter makes a whole Tamil word wrong, so word error rates run much higher than character error rates. Read speech is the easy case. Phone audio, background noise and regional speech come next.

**Kathbath is home ground for the two Tamil systems.** AI4Bharat made Kathbath, and both systems come from AI4Bharat or its partners, so they may have trained on Kathbath's training half. The two tie on Kathbath, though they agree word for word on only 30% of clips.

**One system isn't built for Tamil.** MiniCPM-o 4.5 is a general voice model with no Tamil speech training. A score over 100% means it wrote more wrong words than the clip has.

**Source.** Uraikal track A1. Whisper was transcribed with [`transcribe.py`](https://github.com/zhagaramlabs/uraikal/blob/main/transcribe.py), the others with their own published code.

## Text

### 2. Tokenizers

**What it measures.** Lower is better. How many tokens a tokenizer spends on each Tamil word. Fewer tokens means cheaper and faster use, and more text fits in a model's context. The web text is 2,000 passages of held-out web Tamil, and the literature text is 1,000 passages from Project Madurai.

| Tokenizer | Vocabulary | Tokens per word, web | Tokens per word, literature | Words cut by the pre-tokenizer | Words cut mid-character |
|---|---:|---:|---:|---:|---:|
| [murai-labs/agaram-2.5b](https://huggingface.co/murai-labs/agaram-2.5b) | 64k | 1.68 | 1.93 | 0.0% | 0.0% |
| [ai4bharat/IndicBERTv2-MLM-only](https://huggingface.co/ai4bharat/IndicBERTv2-MLM-only) | 250k | 1.76 | 2.08 | 0.0% | 0.0% |
| [google/muril-base-cased](https://huggingface.co/google/muril-base-cased) | 197k | 1.89 | 2.21 | 0.0% | 0.0% |
| [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) | 251k | 2.15 | 2.41 | 0.0% | 0.1% |
| [google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it) | 262k | 2.44 | 2.69 | 0.0% | 0.0% |
| [sarvamai/sarvam-30b](https://huggingface.co/sarvamai/sarvam-30b) | 262k | 2.44 | 2.69 | 0.0% | 0.0% |
| [sarvamai/sarvam-1](https://huggingface.co/sarvamai/sarvam-1) | 68k | 2.46 | 2.77 | 0.0% | 0.3% |
| [FacebookAI/xlm-roberta-base](https://huggingface.co/FacebookAI/xlm-roberta-base) | 250k | 2.48 | 2.79 | 0.0% | 4.6% |
| [google/mt5-base](https://huggingface.co/google/mt5-base) | 250k | 2.58 | 2.72 | 0.0% | 13.3% |
| [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 200k | 3.19 | 3.41 | 0.0% | 2.2% |
| [mistralai/Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407) | 131k | 3.77 | 3.89 | 0.0% | 5.9% |
| [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) | 248k | 4.28 | 4.34 | 0.0% | 3.2% |
| [deepseek-ai/DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) | 129k | 4.96 | 4.94 | 0.0% | 21.8% |
| [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | 152k | 9.99 | 10.02 | 98.8% | 99.5% |

**The two kinds of cut.** A pre-tokenizer cut splits a letter from its vowel sign before the tokenizer has learned anything. A mid-character cut ends a token inside one character's bytes. Both are always defects.

**Read more.** [Where open tokenizers break Tamil](/papers/tamil-tokenizers/), our draft paper on these numbers.

**Source.** Uraikal track D4, [`tokenizer_eval.py`](https://github.com/zhagaramlabs/uraikal/blob/main/src/uraikal/tokenizer_eval.py).

## Classics

### 3. Does a language model know the Thirukkural?

**What it measures.** Higher is better, except for saying NONE to a real Kural. Each model is asked, in Tamil and English:
- to write a Kural, given its number (200 Kurals);
- to give a Kural's number, given its text (200 more);
- whether a couplet is a valid குறள் வெண்பா (100 couplets, half with their first line's words reversed);
- for a Kural that doesn't exist: a number past 1,330, or one about mobile phones, cricket or tomatoes (40 requests). The right answer is NONE.

| Model | Quoted exactly | Said NONE to a real Kural | Gave its number | Right chapter | Judged the metre | Refused to invent |
|---|---:|---:|---:|---:|---:|---:|
| [GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) | 0.0% | 0.0% | 0.5% | 1.5% | 40.0% | 17.5% |
| [Qwen3 8B](https://huggingface.co/Qwen/Qwen3-8B) | 0.0% | 59.5% | 0.0% | 0.0% | 36.0% | 100.0% |
| [Qwen3.5 9B](https://huggingface.co/Qwen/Qwen3.5-9B) | 0.0% | 39.0% | 0.0% | 0.0% | 36.0% | 100.0% |
| [Sarvam-30B](https://huggingface.co/sarvamai/sarvam-30b) | 0.0% | 60.5% | 0.5% | 1.0% | 36.0% | 80.0% |

**How to read it.** No model could quote a single Kural, and almost none could name a Kural's number. All four answered no to nearly every metre question. Saying NONE is allowed for every request, so a model can't score on invented verses just by refusing everything: the second column shows how often it refused a real Kural. Our [prosody checker](/tools/#3) decides which couplets are valid. About two in three are, because reversing a line's words doesn't always break the metre, so a model that always answers yes scores 64% on metre.

**How we ran it.** Every model ran on a Mac, quantised to 4 bits (Q4_K_M), with thinking turned off and greedy decoding. Quantising can cost a model some recall. Replies stop at 300 tokens. GLM-4.7-Flash often repeats one phrase until then, and Sarvam-30B adds an explanation after its answer; the grader reads the answer. The test items stay private until Uraikal v1, so models can't be trained on them.

**Source.** Uraikal track I, [`classics.py`](https://github.com/zhagaramlabs/uraikal/blob/main/src/uraikal/classics.py).

## Coming next

- **Speech over phone lines and noise:** the same clips passed through phone codecs and mixed with noise.
- **More models on the Thirukkural,** including Gemma 3, Llama 3.1 and Aya.
- **Our own models,** as they ship, scored the same way.

## Add your system

Score your system with the steps in the [Uraikal README](https://github.com/zhagaramlabs/uraikal#run-it), then [open an issue](https://github.com/zhagaramlabs/uraikal/issues) with the results file it writes. We rerun every submission before adding it.