Positionsembeddingar

Positionsembeddingar are vector representations added to token embeddings to inject information about the order of tokens in a sequence. They address the fact that attention mechanisms alone are permutation-invariant, so position information is needed for models to learn syntax and structure. Positionsembeddingar enable transformers and related architectures to differentiate between tokens based on where they appear in the input.

There are several approaches to implementing positionsembeddingar. Absolute position embeddings assign a distinct vector to each position in a sequence, usually as a learned parameter matrix or as a fixed sinusoidal encoding. Learned absolute embeddings are trained like other model parameters, while sinusoidal (fixed) encodings use mathematical functions of the position, enabling some generalization to longer sequences than seen during training. Relative position encodings encode the distance or relationship between pairs of tokens, often by biasing attention scores according to their relative positions. This approach can better capture dependencies between distant tokens and is used in several transformer variants.

Rotary position embeddings (RoPE) are another variant that rotates query and key vectors in attention by an angle determined by position, providing a continuous notion of position without adding a separate embedding vector. In vision transformers, 2D axial or grid-based position embeddings extend the idea to spatial dimensions, assigning separate embeddings along height and width.

Positionsembeddingar influence model capacity and generalization. Absolute methods are simple and widely used, while relative or rotary methods can improve handling of long-range dependencies. The choice depends on the task, data characteristics, and sequence lengths involved.