Sequence models Introduction
Attention
Build an output by weighting information from other positions.
Before this
The intuition
Attention compares a query with keys to assign scores, converts those scores to weights, and combines the corresponding values. This allows different inputs to draw on different context.
A concrete example
If two values are 10 and 20 and their attention weights are 0.25 and 0.75, the weighted output is 17.5.
What to remember
Attention weights describe a computation; they are not automatically a reliable explanation of a model’s decision.
Check your understanding
What output do equal weights give for values 10 and 20?
Reveal explanation
15, their average.