stochasticgradient

Stochastic gradient refers to an estimate of the gradient of a loss function computed from a random subset of data rather than the full dataset. It is a key concept in stochastic gradient descent and related optimization methods, where computing the exact gradient over all training examples can be prohibitively expensive for large-scale problems.

In practice, an iteration updates model parameters by moving in the direction opposite to the estimated gradient. At iteration t, a sample or mini-batch of size B is chosen, the gradient g_t is computed on that subset, and parameters θ are updated as θ ← θ − η_t g_t, where η_t is the learning rate. The expectation of g_t equals the true gradient with respect to the full dataset, assuming unbiased sampling, but g_t introduces stochastic noise that can cause the trajectory to fluctuate.

Mini-batch variants balance computational efficiency and gradient variance: smaller batches yield noisier updates but faster iterations, while larger batches provide more accurate gradient estimates at higher per-iteration cost. Stochastic gradient methods are widely used in machine learning, particularly for training large neural networks and other models where the dataset is too large to process in one pass.

Convergence properties depend on the problem class and learning-rate strategy. For convex objectives, stochastic gradient methods with appropriate diminishing learning rates can converge to a minimum with high probability. In non-convex settings, common in deep learning, they often converge to local minima or saddle points, with stochasticity sometimes helping to escape undesirable points.

Variants and related methods include momentum-based updates and adaptive learning-rate schemes (such as RMSprop, Adagrad, and Adam), which operate on stochastic gradients to improve convergence speed and robustness. Proper data shuffling, scaling, and regularization remain important for effective performance.