självuppmärksamhetsmekanismen

Självuppmärksamhet, often translated as self-attention, is a mechanism used in neural networks to compute representations of a sequence by weighing the relevance of all positions within the same sequence. In this approach, each element in the input sequence is projected into three vectors: a query, a key, and a value. The model computes attention weights by measuring the similarity between the query of a position and the keys of all positions, typically using a scaled dot-product. These weights are then applied to the corresponding values to form a new representation for each position. This allows the model to aggregate information from other parts of the sequence, regardless of distance.

A common extension is multi-head self-attention, where the input is projected into multiple subspaces and attention is performed in parallel across several attention heads. Each head produces its own set of representations, which are then concatenated and projected to form the final output. This enables the model to capture different types of relationships and patterns within the data.

Self-attention is a core component of the Transformer architecture, introduced in the 2017 paper Attention Is All You Need. Transformners use self-attention to process entire sequences in parallel, rather than sequentially as in recurrent networks, which improves efficiency and enables modeling of long-range dependencies. Since then, self-attention has become foundational in many large-scale natural language processing models and has been extended to other domains, including vision and audio.

Applications include language modeling, machine translation, and text generation, as well as vision transformers and multimodal models. Limitations include quadratic growth in memory and computation with sequence length and the need for positional encoding to preserve order information. Researchers address these through various approximate or sparse attention techniques.

See also: attention mechanism; Transformer; multi-head attention.