Self-Attention (சுய-கவனம் / Context Awareness)

Let's find the real meaning of the sentence! (வாக்கியத்தோட அர்த்தத்தை கண்டுபிடிப்போமா!)

Technical Meaning: சுய-கவனம் (Suya-Gavanam) - The mechanism where a model looks at all words in a sentence to understand context.

The Core Idea

Self-Attention is the specific mechanism inside a Transformer that allows a model to weigh the importance of different words within the same sentence. When reading a word, the model dynamically checks every other word in the sentence to understand the grammatical and contextual relationship, solving the problem of ambiguity in human language.

The Origin Story

Traditional models struggled with words that have multiple meanings based on context (e.g., "The bark of the dog" vs "The bark of the tree"). Self-Attention was introduced to allow the word "bark" to pay attention to the word "dog" in the first sentence, and pay attention to the word "tree" in the second sentence. It essentially calculates a mathematical score of relevance between every pair of words in the sequence.

The Tamil Analogy

When translating or understanding a complex sentence, humans naturally do Self-Attention without realizing it.

Look at this sentence: "He went to the bank of the river to deposit his money, but the bank was closed."

• When you read the first "bank", your brain subconsciously pays attention to the word "river" next to it, and you instantly know it means a riverbed (ஆற்றங்கரை).
• When you read the second "bank", your brain pays attention to the word "money" earlier in the sentence, and you instantly know it means a financial institution (வங்கி).

Self-Attention is the mathematical formula that forces the AI to draw these exact same mental connections between words in a sentence, calculating a high "attention score" between "bank" and "money".

Try It Yourself