cuDNN

cuDNN, or CUDA Deep Neural Network library, is a GPU-accelerated library of primitives for deep neural networks developed by NVIDIA. It sits within the CUDA ecosystem and provides optimized routines designed to speed up both training and inference on NVIDIA GPUs.

The library focuses on common neural network operations, including forward and backward passes for convolutional layers, pooling, activation functions, normalization, and recurrent units such as LSTM. It supports multiple data types (FP32, FP16, BF16, INT8) and leverages tensor cores on supported hardware to improve throughput. cuDNN exposes a low-level API that enables frameworks to implement high-performance neural networks while keeping performance portable across different GPU architectures.

Developers and deep learning frameworks access cuDNN through a handle-based API with descriptors for tensors, convolutions, and other primitives. The API provides algorithms for forward and backward convolution and pooling operations, with auto-tuning and heuristic selection to pick efficient paths for a given layer configuration and GPU. It is designed to work within one or more CUDA streams and to interoperate with other CUDA libraries in the NVIDIA stack.

cuDNN is released by NVIDIA as part of the CUDA toolkit and is intended for use on CUDA-enabled NVIDIA GPUs. Compatibility is tied to specific CUDA and driver versions; frameworks regularly update to new cuDNN releases to gain performance and feature improvements. It is widely integrated into major deep learning frameworks such as TensorFlow, PyTorch, and MXNet to accelerate common neural network layers.

As a proprietary library, cuDNN is distributed under NVIDIA's licensing terms and is typically obtained from NVIDIA's developer downloads page, often alongside CUDA toolkits. Users must ensure compliance with licensing and version compatibility when distributing applications that rely on cuDNN.