Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding

2026-09-01

AI Summary

为解决最小贝叶斯风险解码中的度量过拟合问题，本文通过奇异值分解法对效用矩阵进行降噪处理，从而提高文本生成质量。

Abstract

Minimum Bayes Risk (MBR) decoding enables high-quality text generation by selecting the hypothesis that maximizes a utility metric over sampled pseudo-references. However, it is highly susceptible to metric overfitting: it can irregularly inflate the chosen utility metric at the direct expense of other unoptimized evaluation metrics. To mitigate this, we introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal. By computing a low-rank approximation via Singular Value Decomposition (SVD) and retaining only the top-$k$ components, we effectively decouple true consensus from metric noise. Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics. Furthermore, we reveal that this denoising is metric-dependent: neural metrics encode a robust low-rank consensus ideal for SVD, whereas surface-level metrics struggle to separate signal from metric noise.

Problem

• Overfitting
• Minimum Bayes Risk Decoding
• Singular Value Decomposition
• Metric Noise

Innovation

• Singular Value Decomposition
• Minimum Bayes Risk Decoding
• Metric Overfitting
• Low-rank Approximation