Tools

What it does

It splits text into the letters a reader sees, and labels each letter’s length for prosody. It also cleans text before you tokenize or deduplicate it. Released

from uraikal.text import clean, letters, length

letters("தமிழ்")                      # ['த', 'மி', 'ழ்']
[length(x) for x in letters("கடல்")]  # ['kuril', 'kuril', 'otru']
clean("ெகாடு")                        # 'கொடு'

Why it matters

To a computer, தமிழ் is 5 characters, not 3 letters, so len() and string slicing cut letters apart. Text converted from old fonts often types a vowel sign before its consonant (ெ before க). clean moves it back.

Source

text.py

What it does

It writes numbers as spoken Tamil words, so a voice says 2300 the way people do. It also turns Tamil number words back into digits, so a model can do the sums. And it reads old Tamil numerals. Released

from uraikal.numbers import from_numerals, speak_digits, to_digits, to_words

to_words(150000)        # 'ஒரு லட்சத்தி அம்பதாயிரம்'
speak_digits("உங்க பில் 2300 ரூபாய், கடைசி தேதி 15")
# 'உங்க பில் ரெண்டாயிரத்தி முன்னூறு ரூபாய், கடைசி தேதி பதினஞ்சு'
to_digits("முன்னூறுல பதினஞ்சு பெர்சென்ட்")  # '300ல 15 பெர்சென்ட்'
from_numerals("௨௱௫")    # 205

Why it matters

Speech models read digits in English, and language models do sums badly from Tamil words. The ending stays on the number: முன்னூறுல becomes 300ல. In the old numerals, ௱ (100) multiplies the digit before it, so ௨௱௫ is 2 × 100 + 5.

Limits

• It writes spoken forms. When it reads, it accepts formal ones too.
• Whole numbers must be below 100 crore.
• Fractions such as ஒன்றரை aren’t read yet.

Source

numbers.py

What it does

It scans Tamil verse the traditional way (அலகிடுதல்). Letters group into அசை, அசை into சீர், and neighbouring சீர் must link by the rules (தளை). It says whether a couplet is a குறள் வெண்பா and, if not, why. Building It’s on GitHub now and goes to PyPI in the next release.

from uraikal.yaappu import alakidu, check_kural

kural = "வேண்டுதல் வேண்டாமை இலானடி சேர்ந்தார்க்கு\nயாண்டும் இடும்பை இல"
[(seer, name) for seer, _, name in alakidu(kural)]
# [('வேண்டுதல்வேண்', 'கூவிளங்காய்'), ('டாமை', 'தேமா'), ('யிலானடி', 'கருவிளம்'),
#  ('சேர்ந்தார்க்', 'தேமா'), ('கியாண்டு', 'புளிமா'), ('மிடும்பை', 'புளிமா'), ('யில', 'மலர்')]

check_kural("அகர முதல எழுத்தெல்லாம் ஆதி\nபகவன் முதற்றே உலகம்")
# ['வுலகம்: புளிமா is not allowed at the end']

Why the சீர் cross words

Traditional scansion reads a couplet as one stream, after the letters change where words meet (புணர்ச்சி). Then it divides the stream into seven சீர். So சேர்ந்தார்க்கு யாண்டும் scans as சேர்ந்தார்க் கியாண்டு. The checker finds this division by itself, and it matches the one in traditional texts that print the Kural split into சீர்.

How well it works

• All 1,330 Kurals scan as குறள் வெண்பா. We used Tamil Wikisource’s text and corrected 9 typing slips, each against a second transcription.
• It rarely mistakes prose for verse. Only 0.35% of random seven-word stretches of web Tamil pass (7 of 2,000).
• One rule is our own finding. ஃ usually acts as a consonant, but four Kurals only scan if it counts as a short letter, so the checker allows both. A prosody scholar hasn’t checked this yet.

Source

yaappu.py

What it does

It measures any Hugging Face tokenizer on Tamil text: tokens per word, and three kinds of damage. Released

# pip install "uraikal[tokenizers]"
from uraikal.tokenizer_eval import load, measure

text = ["தமிழ் மொழி நாட்டில் பேசப்படுகிறது"]
measure(load("Qwen/Qwen3-8B"), text)      # tokens_per_word 9.0, byte_cut_words 1.0
measure(load("Qwen/Qwen3.5-0.8B"), text)  # tokens_per_word 4.0, byte_cut_words 0.0

The three kinds of damage:

• Letter cut: a token boundary between a consonant and its vowel sign. It isn’t always wrong, since Tamil suffixes often start inside a letter.
• Pre-tokenizer cut: the letter is cut before the tokenizer has learned anything. This is always a defect.
• Byte cut: a token ends in the middle of a character. This is always a defect.

Our results

We measured 14 open tokenizers on 2,000 passages of web Tamil. Read the draft paper, Where open tokenizers break Tamil.

Source

tokenizer_eval.py

What it does

It scores a speech recognizer on Tamil. It reports the share of words wrong (WER) and of characters wrong (CER) for each test set, after a fixed normalizer, so scores from different systems are comparable. Released

from uraikal.score import error_rates
from uraikal.text import normalize

# Whisper wrote விலையாட்டு for விளையாட்டு, sport
error_rates([(normalize("நான் விளையாட்டு பார்த்தேன்"), normalize("நான் விலையாட்டு பார்த்தேன்"))])
# {'wer': 0.333..., 'cer': 0.038...}

One wrong letter makes a whole word wrong. That’s why we report both rates.

To score a whole test set, write your system’s output as one line of JSON per clip, then run:

uraikal score --manifest a1_fleurs.jsonl --hyp my-model.jsonl --system my-model

Source

score.py