randomforest

Random forest is an ensemble learning method for classification and regression that builds a multitude of decision trees and aggregates their predictions. It relies on bootstrap aggregating (bagging) and random feature selection to create diverse trees, reducing variance and the risk of overfitting.

During training, each tree is grown on a bootstrap sample of the training data. At each split, a random subset of features is considered, which decorrelates the trees. Predictions for classification are made by majority vote, while regression uses the average of the trees. Out-of-bag samples—data not included in a given bootstrap sample—provide an internal estimate of generalization error.

Random forests handle high-dimensional data and mixed feature types, are relatively robust to noise and outliers, and can handle missing values in some implementations. They also provide measures of feature importance and can be used for implicit feature selection.

Common applications span many domains, including credit scoring, bioinformatics, marketing analytics, and general predictive modeling on tabular datasets.

Advantages include strong predictive performance, reduced overfitting relative to a single decision tree, and little need for extensive data preprocessing. They offer interpretable metrics such as feature importance but are less transparent than a single tree. Limitations include lower interpretability as a whole, higher computational cost and memory usage, potential bias in importance measures toward certain features, and limited extrapolation beyond the training data.

History: Random forest was introduced by Leo Breiman in 2001. Variants include Extremely Randomized Trees (ExtraTrees) and other ensemble approaches.