Revisiting Meta-Learning with Noisy Labels: Reweighting Dynamics and Theoretical Guarantees

2025-10-14

AI Summary

Over-parameterized models under label noise tend to memorize incorrect labels; existing meta-learning-based sample reweighting methods, though empirically effective, lack theoretical foundations and rely on computationally expensive bilevel optimization. Method: We establish the first three-stage theoretical framework—“warm-up → calibration → post-filtering”—characterizing meta-reweighting training dynamics, revealing a critical noise-sensitivity flaw in the post-filtering stage. To address this, we propose a lightweight alternative that avoids bilevel optimization entirely: it enhances feature discriminability via signal coupling and suppresses noisy gradients through loss contraction. Our method requires only a small clean subset and integrates mean-centering, row-wise shifting, and label-sign modulation. Contribution/Results: On both synthetic and real-world noisy benchmarks, our approach significantly outperforms state-of-the-art reweighting and sample-selection methods, achieving superior robustness, training stability, and computational efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Metareasoning and Metaheuristics
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Learning with noisy labels remains challenging because over-parameterized networks memorize corrupted supervision. Meta-learning-based sample reweighting mitigates this by using a small clean subset to guide training, yet its behavior and training dynamics lack theoretical understanding. We provide a rigorous theoretical analysis of meta-reweighting under label noise and show that its training trajectory unfolds in three phases: (i) an alignment phase that amplifies examples consistent with a clean subset and suppresses conflicting ones; (ii) a filtering phase driving noisy example weights toward zero until the clean subset loss plateaus; and (iii) a post-filtering phase in which noise filtration becomes perturbation-sensitive. The mechanism is a similarity-weighted coupling between training and clean subset signals together with clean subset training loss contraction; in the post-filtering regime where the clean-subset loss is sufficiently small, the coupling term vanishes and meta-reweighting loses discriminatory power. Guided by this analysis, we propose a lightweight surrogate for meta-reweighting that integrates mean-centering, row shifting, and label-signed modulation, yielding more stable performance while avoiding expensive bi-level optimization. Across synthetic and real noisy-label benchmarks, our method consistently outperforms strong reweighting/selection baselines.

Problem

• Analyzing meta-learning sample reweighting dynamics under noisy labels
• Proposing lightweight surrogate to avoid expensive bi-level optimization
• Improving noisy-label learning stability and performance across benchmarks

Innovation

• Meta-reweighting uses clean subset to guide training
• Proposes lightweight surrogate avoiding bi-level optimization
• Integrates mean-centering, shifting, and label-signed modulation