InLine.

Attention Is All You Need

Vaswani et al. · 2017 · arXiv:1706.03762 ↗
Short excerpt & reading notes

“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms” […]

Attention computation

Attention compares each query with a set of keys to determine how much weight to assign to the corresponding values. These scores are normalized before the values are combined:

The matrices Q, K, and V represent queries, keys, and values.The matrices Q, K, and V denote the query, key, and value matrices. The key dimension determines the scaling factor in the denominator. Scaling moderates the dot products before softmax, helping preserve useful gradients.

Reading note

Each head looks at the same information.Each head can capture different relationships. In practice, separate learned projections allow attention heads to represent different relationships. Their outputs are concatenated and projected to produce a combined representation.

Together, these projections provide complementary views of the same sequence.