ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning

2025-08-27

AI Summary

In multi-turn dialogue fine-tuning, low-quality supervision signals—particularly erroneous responses in early turns—induce error propagation, degrading response coherence and quality. Existing static pre-filtering approaches decouple quality control from training, failing to mitigate turn-level error accumulation. This paper proposes a dynamic supervision reliability calibration framework: leveraging Welford’s online algorithm to estimate per-turn loss distributions in real time, it adaptively reweights sample losses without explicit data filtering, enabling fine-grained, turn-level, in-training calibration of supervision quality. Evaluated on both single-source and mixed-quality dialogue datasets, the method significantly improves training stability and response quality. Notably, response scores exhibit a strong positive Spearman correlation with sample size (ρ > 0.8), demonstrating its effectiveness and robustness across diverse data regimes.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Fine-tuning multi-turn dialogue systems requires high-quality supervision but often suffers from degraded performance when exposed to low-quality data. Supervision errors in early turns can propagate across subsequent turns, undermining coherence and response quality. Existing methods typically address data quality via static prefiltering, which decouples quality control from training and fails to mitigate turn-level error propagation. In this context, we propose ReSURE (Regularizing Supervision UnREliability), an adaptive learning method that dynamically down-weights unreliable supervision without explicit filtering. ReSURE estimates per-turn loss distributions using Welford's online statistics and reweights sample losses on the fly accordingly. Experiments on both single-source and mixed-quality datasets show improved stability and response quality. Notably, ReSURE enjoys positive Spearman correlations (0.21 ~ 1.0 across multiple benchmarks) between response scores and number of samples regardless of data quality, which potentially paves the way for utilizing large-scale data effectively. Code is publicly available at https://github.com/Elvin-Yiming-Du/ReSURE_Multi_Turn_Training.

Problem

• Addresses supervision error propagation in multi-turn dialogue fine-tuning
• Mitigates performance degradation from low-quality dialogue training data
• Dynamically down-weights unreliable supervision without explicit filtering

Innovation

• Adaptive learning method for unreliable supervision
• Dynamically down-weights unreliable supervision without filtering
• Uses Welford's online statistics for per-turn loss estimation