Fidelity-Diversity Metrics for Text

2026-07-05

AI Summary

Existing data quality evaluation methods struggle to disentangle and separately measure the fidelity and diversity of textual data, hindering the construction of high-quality training sets. This work addresses this limitation by introducing optimal transport theory into discrete text evaluation for the first time, proposing a pair of metrics based on optimal transport divergences to quantify, respectively, the fidelity (similarity) and diversity (coverage breadth) of candidate texts relative to reference data. The resulting two-dimensional framework effectively reveals the distinct impacts of these qualities on downstream model performance. Experiments on the M2D2 and GSM8K mathematical datasets demonstrate that the proposed metrics accurately identify diversity deficiencies in synthetic data and uncover a significant correlation between such deficiencies and degraded fine-tuned model accuracy.

Abstract

As language modeling technology matures, there is an increasing research focus on the composition and curation of datasets used to train these models. For instance, practitioners commonly seek to augment high-quality datasets with additional text to enhance the performance of models trained on that data. However, informed decisions about data augmentation require more nuanced assessments about data quality. We build on work measuring the precision and recall of generative models to develop a pair of metrics that quantify (1) fidelity, capturing how closely candidate text resembles reference data, and (2) diversity, capturing how well it covers the modes of the reference dataset. Our metrics are based on optimal transport divergence functionals between discrete text summaries. In experiments on M2D2 text datasets, we show that these metrics are able to disentangle a lack of fidelity from a lack of diversity in deficient candidate text. In further experiments, our metrics detect diversity deficits in synthetic GSM8K-style math datasets, which correlate with degradations in downstream accuracy of language models finetuned on this synthetic data.

Problem

• fidelity
• diversity
• text datasets
• data quality
• language modeling

Innovation

• fidelity
• diversity
• optimal transport
• text metrics
• data curation