How to Learn in a Noisy World? Self-Correcting the Real-World Data Noise on Machine Translation

2024-07-02

arXiv.org

AI Summary

To address semantic misalignment noise—i.e., sentence-level alignment with token-level semantic mismatch—prevalent in web-crawled parallel corpora, this paper proposes a self-correcting training paradigm that requires neither external annotations nor aggressive pre-filtering. The core innovation is the first introduction of a token-level self-correction mechanism: leveraging the dynamic growth of model self-knowledge confidence during training, it progressively reweights labels and updates soft targets, using semantic similarity–driven noise simulation as a reference. This approach significantly enhances robustness of neural machine translation under high-noise conditions, yielding BLEU improvements of 2.1–3.8 points on the WMT real-world noisy dataset. Crucially, gains remain stable across varying severities of misalignment, consistently outperforming state-of-the-art denoising methods.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Neuro-Symbolic Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The massive amounts of web-mined parallel data contain large amounts of noise. Semantic misalignment, as the primary source of the noise, poses a challenge for training machine translation systems. In this paper, we first introduce a process for simulating misalignment controlled by semantic similarity, which closely resembles misaligned sentences in real-world web-crawled corpora. Under our simulated misalignment noise settings, we quantitatively analyze its impact on machine translation and demonstrate the limited effectiveness of widely used pre-filters for noise detection. This underscores the necessity of more fine-grained ways to handle hard-to-detect misalignment noise. With an observation of the increasing reliability of the model's self-knowledge for distinguishing misaligned and clean data at the token level, we propose self-correction, an approach that gradually increases trust in the model's self-knowledge to correct the training supervision. Comprehensive experiments show that our method significantly improves translation performance both in the presence of simulated misalignment noise and when applied to real-world, noisy web-mined datasets, across a range of translation tasks.

Problem

• Addressing noise in web-mined parallel data
• Improving machine translation system training
• Implementing self-correction for data noise

Innovation

• Simulated semantic misalignment noise
• Token-level model self-knowledge
• Gradual self-correction training supervision