Self-Error Adjustment: Theory and Practice of Balancing Individual Performance and Diversity in Ensemble Learning

2025-08-06

AI Summary

Addressing the challenge of jointly optimizing individual accuracy and diversity in ensemble learning, this paper proposes the Self-Error Adjustment (SEA) framework. SEA rigorously decomposes ensemble error into two orthogonal, independently controllable components: “self-error” (capturing base learner performance) and “interaction-error” (quantifying diversity), enabling end-to-end trade-off optimization via tunable parameters in the loss function. Theoretically, SEA yields a tighter generalization error bound and a broader adjustment range than conventional approaches. Methodologically, it is task-agnostic—natively supporting both classification and regression without task-specific architectural modifications. Extensive experiments across multiple benchmark datasets demonstrate that SEA consistently outperforms strong baselines, including NCL. Ablation studies further confirm its fine-grained controllability and training stability.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Ensemble learning boosts performance by aggregating predictions from multiple base learners. A core challenge is balancing individual learner accuracy with diversity. Traditional methods like Bagging and Boosting promote diversity through randomness but lack precise control over the accuracy-diversity trade-off. Negative Correlation Learning (NCL) introduces a penalty to manage this trade-off but suffers from loose theoretical bounds and limited adjustment range. To overcome these limitations, we propose a novel framework called Self-Error Adjustment (SEA), which decomposes ensemble errors into two distinct components: individual performance terms, representing the self-error of each base learner, and diversity terms, reflecting interactions among learners. This decomposition allows us to introduce an adjustable parameter into the loss function, offering precise control over the contribution of each component, thus enabling finer regulation of ensemble performance. Compared to NCL and its variants, SEA provides a broader range of effective adjustments and more consistent changes in diversity. Furthermore, we establish tighter theoretical bounds for adjustable ensemble methods and validate them through empirical experiments. Experimental results on several public regression and classification datasets demonstrate that SEA consistently outperforms baseline methods across all tasks. Ablation studies confirm that SEA offers more flexible adjustment capabilities and superior performance in fine-tuning strategies.

Problem

• Balancing individual accuracy and diversity in ensemble learning
• Overcoming limitations of traditional methods like Bagging and Boosting
• Providing tighter theoretical bounds for adjustable ensemble methods

Innovation

• Decomposes ensemble errors into performance and diversity
• Introduces adjustable parameter for precise control
• Provides tighter theoretical bounds and flexibility