Optimizers

An optimizer is an algorithm designed to adjust the parameters of a model to minimize or maximize an objective function, commonly a loss function in machine learning. In mathematical optimization, optimizers search for a parameter vector that decreases the objective, possibly under constraints. They differ in how they use information about the function’s shape, how they handle non-convexity and noise, and how demanding they are in computation and memory.

Gradient-based optimizers use information from derivatives. Basic gradient descent updates parameters in the opposite direction of the gradient. Stochastic gradient descent (SGD) processes small subsets of data and is widely used for large datasets. Mini-batch SGD sits between these extremes. Momentum-based variants, such as SGD with momentum and Nesterov accelerated gradient, accumulate past gradients to accelerate convergence. Adaptive learning rate methods, including Adagrad, RMSprop, Adam, AdaDelta and AMSGrad, adjust step sizes per parameter to improve performance on sparse or noisy problems.

Second-order and quasi-Newton optimizers, like Newton’s method, BFGS, and L-BFGS, use approximate curvature information to potentially converge faster but incur higher computational cost and memory usage, making them common more in traditional optimization than deep learning.

In machine learning practice, the choice of optimizer interacts with model architecture, data, and regularization. Learning rate, momentum, decay, and weight decay are common hyperparameters. Some problems favor robust but slower methods (e.g., SGD with momentum), others benefit from fast adaptive methods (e.g., Adam), while others may require constrained or proximal variants to handle regularization.

Overall, optimizers aim to reach good optima efficiently while managing resources and stability in training, particularly in high-dimensional, noisy, and non-convex landscapes.