Attention is the calculation a language model uses to decide which other words matter for the current word.
Example
In “The bank by the lake was wet”, the model pays attention to “lake” and understands it is a bench by the water, not a financial bank.
For pros
Each token forms query, key and value vectors; their similarity produces weights over all other tokens. Several attention heads work in parallel. The cost grows quadratically with context length, which is why more efficient variants exist.
