Gradientfree

Gradientfree refers to optimization methods that do not require gradient information from the objective function. In gradient-based optimization, the gradient indicates the direction of fastest ascent or descent, guiding the search. Gradientfree or derivative-free optimization relies on function evaluations alone or on surrogate models to locate optima, making it suitable for problems where derivatives are unavailable, expensive to compute, or unreliable due to noise or non-differentiability.

Common approaches fall into several categories. Direct search methods, such as pattern search and certain forms of the Nelder–Mead algorithm, explore the space using simple trial moves without gradient calculations. Stochastic and population-based methods include genetic algorithms and evolution strategies, which explore and combine candidate solutions. Model-based or surrogate methods build inexpensive approximations of the objective, using them to guide sampling; Bayesian optimization and radial basis function surrogates are well-known examples. In some settings, finite-difference or other numerical schemes approximate gradients, but these are still considered gradientfree when gradient information is not assumed or available.

Applications of gradientfree optimization arise in situations with expensive, noisy, discrete, or non-differentiable objective functions. It is commonly used for hyperparameter tuning in machine learning, engineering design, simulations, and other black-box optimization problems where gradient information is impractical to obtain.

Limitations include potentially high function evaluation costs, sensitivity to dimensionality, and lack of guaranteed global optimality. Performance often depends on problem structure, initialization, and the chosen class of algorithm. Gradientfree optimization is frequently described as derivative-free optimization (DFO) in the mathematical optimization literature.