Leaderboard

What it measures

Lower is better. The share of words a system writes wrong, with the share of characters wrong in brackets. Every system is scored after the same Tamil-aware cleanup of the text. FLEURS and Kathbath are public test sets of people reading sentences aloud. Kathbath’s new speakers are people missing from its training data.

How to read it

One wrong letter makes a whole Tamil word wrong, so word error rates run much higher than character error rates. Read speech is the easy case. Phone audio, background noise and regional speech come next.

Kathbath is home ground for the two Tamil systems. AI4Bharat made Kathbath, and both systems come from AI4Bharat or its partners, so they may have trained on Kathbath’s training half. The two tie on Kathbath, though they agree word for word on only 30% of clips.

One system isn’t built for Tamil

MiniCPM-o 4.5 is a general voice model with no Tamil speech training. A score over 100% means it wrote more wrong words than the clip has.

Source

Uraikal track A1. Whisper was transcribed with transcribe.py, the others with their own published code.

What it measures

Lower is better. How many tokens a tokenizer spends on each Tamil word. Fewer tokens means cheaper and faster use, and more text fits in a model’s context. The web text is 2,000 passages of held-out web Tamil, and the literature text is 1,000 passages from Project Madurai.

The two kinds of cut

A pre-tokenizer cut splits a letter from its vowel sign before the tokenizer has learned anything. A mid-character cut ends a token inside one character’s bytes. Both are always defects.

Read more

Where open tokenizers break Tamil, our draft paper on these numbers.

Source

Uraikal track D4, tokenizer_eval.py.

What it measures

Higher is better, except for saying NONE to a real Kural. Each model is asked, in Tamil and English:

• to write a Kural, given its number (200 Kurals);
• to give a Kural’s number, given its text (200 more);
• whether a couplet is a valid குறள் வெண்பா (100 couplets, half with their first line’s words reversed);
• for a Kural that doesn’t exist: a number past 1,330, or one about mobile phones, cricket or tomatoes (40 requests). The right answer is NONE.

How to read it

No model could quote a single Kural, and almost none could name a Kural’s number. All four answered no to nearly every metre question. Saying NONE is allowed for every request, so a model can’t score on invented verses just by refusing everything: the second column shows how often it refused a real Kural. Our prosody checker decides which couplets are valid. About two in three are, because reversing a line’s words doesn’t always break the metre, so a model that always answers yes scores 64% on metre.

How we ran it

Every model ran on a Mac, quantised to 4 bits (Q4_K_M), with thinking turned off and greedy decoding. Quantising can cost a model some recall. Replies stop at 300 tokens. GLM-4.7-Flash often repeats one phrase until then, and Sarvam-30B adds an explanation after its answer; the grader reads the answer. The test items stay private until Uraikal v1, so models can’t be trained on them.

Source

Uraikal track I, classics.py.