FP16

FP16, or half-precision floating point, is a 16-bit format defined by IEEE 754-2008 (binary16). It uses 1 sign bit, 5 exponent bits, and 10 fraction bits, with a bias of 15. Exponent field values 1–30 denote normal numbers; 0 denotes zero or subnormal, and 31 denotes infinity or NaN. The maximum finite normal value is (2−2^−10)×2^15 (about 6.5504×10^4); the smallest positive normal is 2^−14. Subnormals fill the range below normal numbers, down to about 5.96×10^−8.

FP16 reduces memory use and can increase throughput on hardware that supports it, at the cost of precision and dynamic range relative to FP32. It has 10 fraction bits, yielding roughly 3–4 decimal digits of precision. This can cause rounding, underflow, or overflow in some computations. Mixed-precision techniques, including loss scaling and using FP32 accumulators, help maintain numerical stability in machine learning.

Hardware and software support: Many GPUs provide native FP16 arithmetic, notably NVIDIA GPUs with Tensor Cores for mixed-precision training. Other vendors also support half precision. CPUs may offer 16-bit types or conversions via SIMD. In software, FP16 is available as data types in NumPy, PyTorch, and TensorFlow, and the IEEE 754 binary16 format is the defined standard.

Applications: FP16 is widely used in deep learning for model inference and training where memory and bandwidth are limiting factors. It is also used in graphics and some signal-processing workflows. For numerically sensitive tasks, FP32 or higher precision is used, or mixed-precision strategies are employed to balance speed and accuracy.