hyperband

Hyperband is a resource allocation framework for hyperparameter optimization that combines random search with principled early stopping. It is designed to identify promising hyperparameter configurations under limited computational budgets by allocating more resources to better-performing configurations and pruning weaker ones early in the evaluation process. The method is model-agnostic and can be applied to any setting where a configuration can be evaluated with a measurable amount of resources, such as training epochs or iterations.

The core idea is to run many configurations at small budgets and progressively increase the budget for the most promising ones. Hyperband orchestrates multiple brackets, each with a different initial number of configurations and a different starting resource level. Within each bracket, a procedure similar to successive halving is used: configurations are trained for a small budget, ranked by performance, and a portion of the worst performers are discarded, after which the remaining configurations receive more resources and are reevaluated. By combining several brackets, Hyperband explores a wide range of configurations and resource allocations, balancing breadth of search with depth of evaluation.

Key parameters include the maximum resource per configuration and a halving factor that controls how aggressively budgets are increased. The method is valued for its efficiency, often delivering strong results with less computation than exhaustive search. It has inspired variants such as BOHB, which integrates Bayesian optimization with Hyperband to guide the sampling of configurations. Limitations include reliance on the premise that performance at small budgets correlates with final performance and sensitivity to noisy or highly variable evaluations. Hyperband has been widely used in tuning neural networks and other machine learning models.