momentummenetelmä

The momentum method is a technique used in optimization and machine learning to accelerate convergence. It is particularly useful for deep neural networks where gradients can be noisy and the optimization landscape can be complex. The core idea behind the momentum method is to incorporate information from past gradients to influence the current update direction. Instead of solely relying on the current gradient, the method adds a fraction of the previous update vector to the current gradient. This fraction is controlled by a momentum coefficient, typically denoted by gamma, which ranges between 0 and 1.

When gamma is close to 0, the momentum method behaves similarly to standard gradient descent, with little influence from past updates. As gamma approaches 1, the influence of past updates becomes stronger, leading to smoother and faster convergence, especially in directions where the gradient has been consistently pointing. This can help the optimizer "roll downhill" more effectively, overcoming small local minima or plateaus.

The mathematical formulation involves maintaining a velocity vector, which is updated at each step. The velocity is a combination of the previous velocity and the current gradient. The parameters are then updated using this velocity vector, scaled by the learning rate. The momentum method helps to dampen oscillations in directions with high curvature and accelerate progress in directions with consistent gradients. It has become a standard component in many advanced optimization algorithms, such as Adam and RMSprop, often in conjunction with other adaptive learning rate techniques.