MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training

2025-12-03

AI Summary

Large-scale code datasets in software engineering commonly contain label noise, severely degrading the robustness and generalization of deep learning models—particularly large language models (LLMs). To address this, we introduce the first systematic application of noisy label learning (NLL) to software engineering, proposing an embedded noise identification and suppression framework. During fine-tuning of pretrained code models, our approach jointly leverages per-sample loss dynamics, adaptive dropout, and Gaussian mixture model clustering to iteratively detect and filter persistent noisy samples across multiple training stages. Crucially, it requires no additional annotation or manual dataset cleaning. Evaluated on code summarization and commit intention classification, our method consistently improves diverse LLMs, yielding average gains of +2.1 BLEU and +3.4 F1. It effectively mitigates label noise interference and establishes a novel paradigm for robust, large-scale code model training in software engineering.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The reliable application of deep learning models to software engineering tasks hinges on high-quality training data. Yet, large-scale repositories inevitably introduce noisy or mislabeled examples that degrade both accuracy and robustness. While Noise Label Learning (NLL) has been extensively studied in other fields, there are a few works that investigate NLL in Software Engineering (SE) and Large Language Models (LLMs) for SE tasks. In this work, we propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process of code-Pretrained Language Models (PTM) and code-LLMs. We first investigate the effect of noise at varying levels on convergence and loss trajectories of the models. Then we apply an adaptive dropout strategy guided by per-sample loss dynamics and Gaussian Mixture Model clustering to exclude persistently noisy points while preserving clean data. Applying to code summarization and commit intent classification, our experiments reveal that some LLMs are more sensitive to noise than others. However, with MANTRA, the performance of all models in both tasks is improved. MANTRA enables researchers and practitioners to reduce the impact of errors introduced by the dataset in training, saves time in data cleaning and processing, while maximizing the effect of fine-tuning.

Problem

• Addresses noisy data in software engineering deep learning training
• Proposes adaptive noise treatment for code language models
• Improves model robustness and reduces data cleaning effort

Innovation

• Multi-stage adaptive noise treatment framework
• Adaptive dropout guided by loss dynamics
• Gaussian Mixture Model clustering for noise exclusion