DistGrv

DistGrv is a class of optimization methods for distributed training that employ gradient variance reduction to accelerate convergence on finite-sum objectives common in machine learning. It extends classic variance-reduced techniques such as SVRG and SAGA to multi-node environments, combining local stochastic gradients with a periodically refreshed reference gradient computed from data distributed across workers.

Conceptually, each worker maintains a local parameter copy and a snapshot of its gradient; at regular intervals a central coordinator (or a distributed consensus step) computes a reference gradient over the full or a representative subset of the data. During subsequent local updates, workers form corrected gradients by adding the difference between the current local gradient and the stored snapshot gradient to the reference gradient, reducing the variance of stochastic updates and improving convergence pace. Updates are aggregated via a parameter server or all-reduce, and can be implemented synchronously or asynchronously.

DistGrv is applicable to problems with smooth finite-sum objectives, including convex and some non-convex models such as deep learning with large datasets. Theoretical results typically show faster convergence rates than naive distributed SGD under standard assumptions; practical performance depends on communication cost, data partitioning, and system latency.

Practical considerations include choosing the update cadence for the reference gradient, balancing computation and communication, and handling stragglers or faults. DistGrv remains an active area of research, with ongoing work exploring adaptive sampling, gradient compression, and hybrid asynchronous schemes to further improve scalability.