Semigradients

Semigradients are a mathematical construct that extends the classical notion of a gradient to functions that may not be differentiable in the traditional sense. Introduced in the mid‑1990s in the context of convex optimization, a semigradient at a point of a real‑valued function provides a linear approximation that is guaranteed to lie below the function in a neighborhood of that point. This property makes semigradients useful for establishing global convergence guarantees in iterative minimization algorithms.

Formally, given a function \(f:\mathbb{R}^n \rightarrow \mathbb{R}\), a vector \(g\) is called a semigradient at \(x_0\) if there exists a neighborhood \(U\) of \(x_0\) such that for all \(x \in U\), \(f(x) \ge f(x_0)+g^{T}(x-x_0)\). This definition is similar to that of a subgradient, but the inequality is required only locally rather than globally. Consequently, every differentiable point has a unique semigradient equal to the usual gradient, while non‑differentiable points may admit multiple semigradients or none at all.

The concept of semigradients has been applied to several algorithmic frameworks, most notably in bundle methods for nonsmooth optimization. By maintaining a collection of semigradients from previous iterations, these methods are able to construct cutting-plane models that approximate the target function from above, thereby guiding the search for a minimizer. In machine learning, semigradients have been used to analyze the behavior of loss functions that exhibit plateau regions or kinks, allowing for adaptive step‑size rules in stochastic gradient descent.

While less widely known than subgradients, semigradients offer a useful intermediate tool for analyzing and solving optimization problems where full subgradient information is either unavailable or insufficiently informative. Researchers continue to investigate their theoretical properties and practical implementations, particularly in large‑scale, high‑dimensional settings.