5. Language Models
Tokenization (துண்டாக்குதல் / Chopping Words)
Let's break down long Tamil words! (தமிழ் வார்த்தைகளை பிரிப்போமா!)
Technical Meaning: வில்லைகளாக்குதல் (Villaigalaakkuthal) - Breaking text down into smaller, manageable chunks (tokens) for the AI to process.
The Core Idea
AI models don't actually read words; they read numbers. Tokenization is the first step of processing language, where a sentence is chopped up into smaller pieces called "Tokens" (which can be whole words, syllables, or even single letters). These tokens are then mapped to specific ID numbers in the AI's dictionary.
The Origin Story
Initially, early AIs treated every single unique word as a token. But human languages are too complex—we add prefixes and suffixes all the time (like "run", "running", "runner"). If an AI treats them all as separate, its dictionary becomes infinitely large. Modern AIs use "Subword Tokenization" (like Byte-Pair Encoding), which breaks complex words down into root components, saving massive amounts of memory.
The Tamil Analogy
Tamil is an agglutinative language, meaning we combine multiple words into one massive word.
Take the word: "வந்துகொண்டிருக்கிறேன்" (Vanthukondirukkiren - "I am coming").
If you ask an AI to memorize every massive Tamil combination, its brain will explode. Instead, it uses Tokenization to chop it up into smaller, meaningful root components:
- வந்து (Vanthu - Come)
- கொண்டு (Kondu - Continuous action)
- இருக்கிறேன் (Irukkiren - I am)
By breaking it down into 3 simple tokens, the AI can easily understand the grammar and reuse these root words to build millions of other sentences!
Try It Yourself
Tokenization
LLMs don't read words. They chop words into smaller sub-word pieces!