autograd

Autograd refers to automatic differentiation systems that compute derivatives of numeric functions automatically. It is particularly important in optimization and machine learning, where gradients guide parameter updates. The term covers several implementations, including the original Python library autograd for NumPy-style code and the differentiation features built into modern frameworks such as PyTorch, TensorFlow, and JAX.

Most autograd systems work by tracing the sequence of arithmetic operations performed during a function's evaluation to build a graph of differentiable operations. When a gradient is requested, reverse-mode automatic differentiation traverses this graph from outputs to inputs, applying the chain rule to accumulate gradients. This approach computes derivatives efficiently for functions with many inputs.

Usage and scope: Autograd libraries support gradient-based optimization, backpropagation in neural networks, and higher-order derivatives. They generally differentiate numeric arrays through differentiable operations, while handling common control flow and in-place operations in specific ways. Limitations include incomplete coverage of arbitrary Python code, potential overhead from tracing, and performance depending on the backend and implementation. Some frameworks also provide higher-order derivatives and vector-Jacobian products.

History and impact: The development of autograd-inspired tools influenced the design of contemporary deep learning frameworks, enabling dynamic graphs and automatic gradient computation. These systems have become central to modern machine learning, supporting tasks from simple optimization to complex neural architectures and research into differentiable programming.