Halvprecision

Halvprecision, or half-precision, is a 16-bit floating-point format used in computer arithmetic. In the IEEE 754 standard, binary16 consists of 1 sign bit, 5 exponent bits, and 10 fraction bits. The exponent has a bias of 15, giving ranges for normal numbers, subnormals, infinities, and NaNs. The machine epsilon, the distance between 1.0 and the next representable number, is 2^-10 ≈ 0.0009766.

The smallest positive normal number is 2^-14, and the smallest positive subnormal is 2^-24. Normal numbers cover roughly from 6.1035×10^-5 up to 6.5504×10^4. Subnormal values extend toward zero, allowing representations closer to zero than the normal range. The format can represent zero, positive and negative infinity, and NaNs.

Halvprecision is often abbreviated FP16 or half-precision and is used to reduce memory usage and bandwidth, especially in graphics pipelines and neural network workloads. Hardware support is common in GPUs and specialized accelerators; software libraries provide conversion to and from FP32, FP64, and other formats.

Limitations include a reduced dynamic range and precision compared with FP32, which can cause numerical errors in accumulation and gradient calculations. To mitigate this, mixed-precision techniques are used, computing in FP16 with FP32 accumulation and sometimes applying loss scaling. Alternatives such as bfloat16 offer a larger exponent range with fewer mantissa bits, trading some precision for range.

See also: IEEE 754, FP32, bfloat16, mixed precision training.