Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity
Ensemble learning is a process by which multiple base learners are strategically generated and combined into one composite learner. There are two features that are essential to an ensemble's performance, the individual accuracies of the component learners and the overall diversity in the ensemble. The right balance of learner accuracy and ensemble diversity can improve the performance of machine learning tasks on benchmark and real-world data sets, and recent theoretical and practical work has demonstrated the subtle trade-off between accuracy and diversity in an ensemble. In this paper, we extend the extant literature by providing a deeper theoretical understanding for assessing and improving the optimality of any given ensemble, including random forests and deep neural network ensembles. We also propose a training algorithm for neural network ensembles and demonstrate that our approach provides improved performance when compared to both state-of-the-art individual learners and ensembles of state-of-the-art learners trained using standard loss functions. Our key insight is that it is better to explicitly encourage diversity in an ensemble, rather than merely allowing diversity to occur by happenstance, and that rigorous theoretical bounds on the trade-off between diversity and learner accuracy allow one to know when an optimal arrangement has been achieved.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityEnsemble LearningSimilar Papers 제목 키워드 기반
Developing parsimonious ensembles using ensemble diversity within a reinforcement learning framework
Heterogeneous ensembles built from the predictions of a wide variety and large number of diverse base predictors represent a potent approach to building predictive models for problems where the ideal base/individual pred…
Diversityreinforcement-learningReinforcement LearningReinforcement Learning (RL)Improving robustness and calibration in ensembles with diversity regularization
Calibration and uncertainty estimation are crucial topics in high-risk environments. We introduce a new diversity regularizer for classification tasks that uses out-of-distribution samples and increases the overall accur…
DiversityOut-of-Distribution DetectionDICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members'…
DiversityOut-of-Distribution DetectionEnhancing Time Series Classification with Diversity-Driven Neural Network Ensembles
Ensemble methods have played a crucial role in achieving state-of-the-art (SOTA) performance across various machine learning tasks by leveraging the diversity of features learned by individual models. In Time Series Clas…
Time Series ClassificationEnsemble LearningPathologies of Predictive Diversity in Deep Ensembles
Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models, e.g. through bagging or boosting. Here we demonstrate that these intuitions do not apply to high-c…
Diversity