Attention Mechanism (கவனம் செலுத்துதல் / Focusing)

Let's manage Chennai traffic! (சென்னை டிராபிக் கண்ட்ரோல் பண்ணுவோமா!)

Technical Meaning: கவன ஈர்ப்பு நுட்பம் (Gavana Eerppu Nutpam) - A technique that tells the AI which parts of the input to focus on.

The Core Idea

The Attention Mechanism is the breakthrough that led to modern AI like ChatGPT. Instead of reading a sentence word-by-word from left to right, an Attention-based model looks at the entire sentence at once, calculating how much "focus" or "attention" each word should pay to every other word to understand the true context.

The Origin Story

Older models (like RNNs) read text sequentially. By the time they reached the end of a long paragraph, they often "forgot" what the first sentence was about. In 2017, Google researchers published the famous paper “Attention Is All You Need,” proving that if every word can instantly reference every other word, the AI never loses the context.

The Tamil Analogy

Think of this like an experienced Chennai Traffic Police Officer (டிராஃபிக் போலீஸ்) at a busy four-way junction like Gemini Flyover.

If he only looked at one car at a time (sequential reading), he would cause an accident. Instead, his eyes dart around, taking in the entire junction simultaneously. He pays 80% of his "attention" to the speeding MTC Bus coming from the left, 15% attention to the auto-rickshaw trying to squeeze through, and 5% to a parked cycle.

He dynamically shifts his attention based on what is most important right now to understand the total state of the traffic. Language models do exactly this with words in a sentence!

Try It Yourself

Hover over the word "bank" below to see how the AI shifts its attention to surrounding words to figure out what "bank" means in different contexts.

Concept 4 · Transformers

Attention is all you need

How does an AI know that 'bank' means a river side in one sentence, but a financial building in another? The Attention mechanism looks at the surrounding context words to figure out the meaning. Hover over the word 'bank' below.

I

sat

by

the

river

bank

to

read

a

book.

Context Switcher

Context AContext B

How it works

Before Transformers, AI read words one by one. Now, the Self-Attention Mechanism allows the AI to look at every single word in the sentence simultaneously to figure out the perfect context for the current word.