TensorRT

TensorRT is a high-performance deep learning inference optimizer and runtime developed by NVIDIA. It is designed to accelerate inference for trained neural networks on NVIDIA GPUs, delivering low latency and high throughput for data center and edge deployments. It supports FP32, FP16, and INT8 precision and is used to deploy models in production environments.

Key components include the parser, which imports networks from common frameworks (notably ONNX as a standard exchange format), an optimizer and builder that perform graph optimizations such as layer fusion, precision calibration for INT8, kernel auto-tuning, and memory optimizations, and the runtime that executes the optimized inference engine. The system can generate a serialized engine that can be loaded for fast startup.

Workflow typically involves exporting a trained model to a compatible format (often ONNX); using the TensorRT parser to read the network; configuring precision modes and calibrators; building the engine; serializing or deserializing the engine; and running inference via the runtime API. INT8 calibration uses a representative dataset to determine scale factors.

TensorRT supports a subset of operators; when a model uses unsupported layers, alternatives include fusing supported layers, using custom plugins, or converting portions to supported ops. It emphasizes compatibility with NVIDIA GPUs and the CUDA ecosystem.

Applications include real-time inference in autonomous vehicles, robotics, medical imaging, and edge AI. It provides APIs in C++ and Python and integrates with other NVIDIA software stacks, including CUDA, cuDNN, and NVIDIA's inference tooling for deployment at scale.

TensorRT is a proprietary library provided by NVIDIA as part of its developer tools. It is distributed with the TensorRT SDK and is commonly used in production AI deployments on NVIDIA hardware.