Loss Functions Robust to the Presence of Label Errors

2025-11-20

AI Summary

To address performance degradation caused by label noise in training data, this paper proposes two novel robust loss functions. Methodologically, the approach adapts the sample reweighting mechanism of Focal Loss but introduces a more refined strategy for identifying and suppressing hard-to-classify instances—thereby reducing overfitting to potentially mislabeled samples. The proposed losses dynamically down-weight gradient contributions from such instances in a self-adaptive manner, without requiring auxiliary modules or prior knowledge of noise rates. Extensive experiments on benchmark datasets with synthetically injected label noise demonstrate that the new losses significantly improve label noise detection accuracy, achieving average F1-score gains of +3.2–5.8 percentage points over standard cross-entropy and Focal Loss. The methods are both computationally lightweight and empirically effective, offering a simple yet powerful alternative for learning under label corruption.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Methods for detecting label errors in training data require models that are robust to label errors (i.e., not fit to erroneously labelled data points). However, acquiring such models often involves training on corrupted data, which presents a challenge. Adjustments to the loss function present an opportunity for improvement. Motivated by Focal Loss (which emphasizes difficult-to-classify samples), two novel, yet simple, loss functions are proposed that de-weight or ignore these difficult samples (i.e., those likely to have label errors). Results on artificially corrupted data show promise, such that F1 scores for detecting errors are improved from the baselines of conventional categorical Cross Entropy and Focal Loss.

Problem

• Developing loss functions robust to label errors in training data
• Addressing model training challenges with corrupted datasets
• Improving detection accuracy for mislabeled samples during training

Innovation

• Novel loss functions de-weight difficult samples
• Loss functions ignore samples with likely label errors
• Improved F1 scores over Cross Entropy and Focal Loss