boosting

Boosting is an ensemble learning technique in machine learning that builds a sequence of weak predictive models and combines them to form a single stronger predictor. The models are trained one after another, with each new model attempting to correct the errors of its predecessors. The final prediction is typically a weighted sum or vote of the individual models.

The most influential boosting algorithms include AdaBoost (Adaptive Boosting), which increases the emphasis on difficult training examples; Gradient Boosting, which fits new models to the residual errors of the current ensemble by optimizing a differentiable loss function; and modern variants such as XGBoost, LightGBM, and CatBoost that add regularization, faster training, and better handling of different data types. Boosting is distinct from bagging, which trains models in parallel and averages their outputs; boosting relies on a sequential dependency where each model focuses on correcting prior mistakes.

How boosting works: you start with a base learner, often a shallow decision tree. In AdaBoost, misclassified examples receive higher weights so the next learner concentrates on them. In gradient boosting, each new learner is trained to predict the negative gradient of the loss function, effectively reducing the overall error. As more trees are added, the ensemble typically reduces bias and improves accuracy, though hyperparameters such as learning rate, number of estimators, and tree depth influence bias-variance trade-offs.

Applications include classification and regression on structured data across domains like finance, marketing, and bioinformatics. Strengths include strong predictive performance and flexibility; weaknesses include sensitivity to noisy data and outliers, potential overfitting without proper regularization, and higher training costs relative to simpler models.