TF32

TF32 is a single-precision floating-point format developed by NVIDIA. It is a 32-bit format that offers a balance between performance and precision for certain types of computations, particularly in graphics processing and deep learning. TF32 is designed to be a drop-in replacement for standard FP32 (single-precision) calculations, meaning that code written for FP32 can often run on hardware supporting TF32 without modification, and will see performance improvements.

The key characteristic of TF32 is its use of the same 8-bit exponent and 1-bit sign as FP32, but with a reduced 10-bit mantissa. This reduction in mantissa bits compared to the standard 23 bits of FP32 allows for faster computation. Despite the reduced mantissa, TF32 retains the same dynamic range as FP32 due to the shared exponent. This makes it suitable for workloads where the full precision of FP32 is not strictly necessary but where the performance gains are significant. TF32 is prominently featured in NVIDIA's Ampere and Hopper GPU architectures, enabling faster training of deep neural networks and improved performance in graphics rendering tasks.