derivativefree

Derivative-free optimization (DFO) refers to a class of optimization techniques designed to minimize or maximize objective functions without requiring gradient or Hessian information. These methods are particularly useful when the objective is non-differentiable, noisy, expensive to evaluate, or available only as a black-box.

DFO methods are broadly categorized into direct-search methods and model-based methods. Direct-search methods explore the search space using patterns or simplex structures, with examples such as the Nelder-Mead simplex method and various pattern-search algorithms. Model-based or surrogate methods build an approximate model of the objective from sample evaluations and guide the search using that model; notable approaches include Gaussian process-based Bayesian optimization, radial-basis-function surrogates, and evolution strategies such as CMA-ES that do not rely on explicit derivatives. Powell's method is another classic, relying on line searches in conjugate directions without gradient computations.

Key considerations in choosing a DFO approach include the cost of objective evaluations, the presence of noise, and the dimensionality of the problem. Convergence guarantees vary; some methods are proven to converge to stationary points under mild assumptions, while global optimality is generally difficult for non-convex problems. High-dimensional problems pose particular challenges, and hybrid methods often combine surrogate modeling with selective gradient information when available.

Applications span engineering design, simulation-based optimization, hyperparameter tuning in machine learning when gradients are unavailable or unreliable, robotics, and materials science. Derivative-free methods are valued for robustness to non-smoothness and for enabling optimization in black-box settings where derivative information cannot be trusted or is impractical to obtain.