metaensemble

A metaensemble is an ensemble learning approach where multiple base models generate predictions that are then combined by a secondary model called a meta-model. The goal is to improve predictive performance by leveraging complementary strengths of diverse learners.

Common procedures include stacking (stacked generalization) and blending. In stacking, predictions of base models are collected on held-out data (out-of-fold) to train the meta-model, which then makes final predictions. Blending uses a holdout set to train the meta-model, often simpler.

Base models can be of different types (e.g., decision trees, logistic regression, neural networks) and the inputs to the meta-model are typically predicted probabilities or raw scores from the base models, optionally plus original features. Meta-models can be linear, tree-based, or neural networks.

Advantages include improved accuracy, robustness, and the ability to combine heterogeneous models. Limitations include risk of overfitting, data leakage if not careful, increased computational cost, and reduced interpretability.

Applications include Kaggle competitions, production forecasting, and any scenario where model diversity can be exploited. Related concepts include ensemble methods, stacking, blending, and cascading ensembles.