SGDs

Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used to minimize differentiable objective functions that can be expressed as a sum over data points. In machine learning, it is commonly applied to train models by minimizing a loss function defined on training examples.

At each iteration, SGD updates the model parameters θ by moving in the direction opposite to the gradient of the loss with respect to θ. When a single example is used, the update is θ := θ − η ∇ℓ(θ; x_i, y_i); with a minibatch B of size m, the update is θ := θ − η (1/m) Σ_{i∈B} ∇ℓ(θ; x_i, y_i). The learning rate η can be fixed or follow a schedule. The stochasticity introduces noise that can help escape shallow minima but can also slow convergence.

Convergence properties depend on the objective: for convex objectives, SGD can converge to a global minimum under appropriate learning rate decay; for non-convex objectives such as deep neural networks, SGD often yields good solutions but does not guarantee a global optimum. Practical use relies on proper tuning of the learning rate, batch size, and data shuffling, sometimes with learning rate schedules or momentum.

Variants and extensions include momentum, Nesterov accelerated gradient, AdaGrad, RMSProp, and Adam. These variants modify how updates are computed, often by adapting learning rates per parameter or incorporating past gradients to stabilize and speed up convergence.

Applications are widespread in training large-scale models, especially neural networks, as well as linear models and other algorithms where the dataset is too large to process in full batch. SGD was introduced by Robbins and Monro in 1951 as a stochastic approximation method.