Self-Attention and Multi-Head Attention Explained Clearly in Generative AI
Self-Attention and Multi-Head Attention Explained Clearly
Self-attention means each word in a sentence looks at all other words to decide how much attention it should pay to them.
1) Query, Key, Value Concept
Each token is converted into three vectors:
- Query (Q)
- Key (K)
- Value (V)
Attention score = similarity between Query and Key. Final output = weighted sum of Value vectors.
2) Why Multi-Head?
A single attention mechanism might focus only on one type of relationship. Multi-head attention allows the model to learn different types of relationships simultaneously.
- Head 1: Grammar relations
- Head 2: Semantic similarity
- Head 3: Positional dependencies
3) Why This Matters for LLMs
Multi-head attention enables richer contextual understanding, which is why large language models generate coherent long responses.
4) Summary
Self-attention allows tokens to understand each other. Multi-head attention allows multiple types of understanding at once.
