Gradientendescent

Gradientendescent, commonly known as gradient descent, is an iterative optimization method for minimizing differentiable functions. Given an objective function f: R^n → R and an initial parameter vector θ0, the algorithm repeats: compute the gradient g_t = ∇f(θ_t); update θ_{t+1} = θ_t − α_t g_t, where α_t is the step size or learning rate. The gradient points in the direction of greatest increase, so the negative gradient moves toward a minimum. Convergence depends on the learning rate and the properties of f; with a fixed, suitably small α, gradient descent can converge on many convex problems, while in non-convex settings it may reach a local minimum or saddle point.

Variants include batch gradient descent, which uses the full dataset to compute the gradient; stochastic gradient descent, which updates from a single example; and mini-batch gradient descent, which uses a subset of data per update. In machine learning, the objective is often a loss function over data, and gradient descent updates model parameters to minimize this loss.

Practical improvements include momentum, Nesterov acceleration, and adaptive learning-rate methods such as AdaGrad, RMSprop, and Adam. Line search and learning-rate schedules can adjust α during training. Key considerations include choosing a reasonable initialization, data scaling, and the problem’s conditioning, as these affect convergence speed and stability. Gradient descent remains a foundational algorithm in optimization and is widely used for training linear models, neural networks, and other differentiable models.