Annealed Multiple Choice Learning: Overcoming limitations of Winner-takes-all with annealing

2024-07-22

arXiv.org

AI Summary

To address the local optima trapping and hypothesis diversity suppression inherent in Winner-takes-all (WTA) strategies within Multiple Choice Learning (MCL), this paper proposes a novel multi-hypothesis learning framework integrating Simulated Annealing (SA). It is the first work to incorporate the annealing mechanism from statistical physics into MCL, dynamically modulating competitive intensity via temperature scheduling to enhance global exploration of the hypothesis space. Theoretical analysis characterizes the training trajectory, while an information-theoretic regularizer—designed to be compatible with WTA—is jointly optimized with the task loss to balance diversity and accuracy. Empirical evaluation on synthetic data, UCI benchmarks, and speech separation tasks demonstrates that the method significantly outperforms standard MCL: it achieves more stable convergence, improves prediction diversity by up to 23.6%, and boosts average accuracy by 1.8–4.2 percentage points.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We introduce Annealed Multiple Choice Learning (aMCL) which combines simulated annealing with MCL. MCL is a learning framework handling ambiguous tasks by predicting a small set of plausible hypotheses. These hypotheses are trained using the Winner-takes-all (WTA) scheme, which promotes the diversity of the predictions. However, this scheme may converge toward an arbitrarily suboptimal local minimum, due to the greedy nature of WTA. We overcome this limitation using annealing, which enhances the exploration of the hypothesis space during training. We leverage insights from statistical physics and information theory to provide a detailed description of the model training trajectory. Additionally, we validate our algorithm by extensive experiments on synthetic datasets, on the standard UCI benchmark, and on speech separation.

Problem

• Multi-choice Learning
• Winner-takes-all Strategy
• Local Optima Problem

Innovation

• Annealing Multiple Choice Learning
• Global Optimum
• Simulated Annealing