The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?

2025-09-26

AI Summary

In class-incremental learning (CIL) evaluation, mainstream protocols estimate performance via mean and variance over only a few randomly sampled class orders—leading to biased mean estimates and severely underestimated variance, thus failing to reflect true generalization capability. This work argues that reliable assessment necessitates comprehensive characterization of the performance distribution. To this end, we propose EDGE, an adaptive evaluation protocol based on extremal sequences. First, we theoretically establish a negative correlation between inter-task similarity and model performance, enabling similarity-driven search for extremal (best/worst-case) class orders. Second, we integrate theoretical analysis with dynamic sampling to construct a robust performance distribution estimator. Experiments demonstrate that EDGE substantially reduces estimation bias in both mean and variance, accurately captures performance bounds, and provides reproducible, high-confidence evaluation for model selection and robustness validation.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Game Theory and Economic Paradigms: Adversarial Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable performance across all possible class sequences. In real-world settings, the order in which classes arrive is diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation protocols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and a severe underestimation of the true variance in the performance distribution. We therefore contend that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role in the reliable evaluation of CIL. Moreover, we observe a consistent positive correlation between inter-task similarity and model performance, a relation that can be leveraged to guide the search for extreme sequences. Building on these insights, we propose EDGE (Extreme case-based Distribution and Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-truth performance distribution. Extensive experiments demonstrate that EDGE effectively captures performance extremes and yields more accurate estimates of distributional boundaries, providing actionable insights for model selection and robustness checking. Our code is available at https://github.com/AIGNLAI/EDGE.

Problem

• Current CIL evaluation underestimates true performance variance across sequences
• Robust evaluation must characterize the full performance distribution accurately
• Proposed EDGE protocol identifies extreme sequences using inter-task similarity

Innovation

• Evaluates class sequences using inter-task similarity
• Adaptively identifies extreme sequences for testing
• Approximates true performance distribution more accurately