Validation-Induced Shapley Shifts: How Validation Structure Distorts Data Valuation

2026-07-03

AI Summary

This work addresses the stability of data valuation via Shapley values, revealing a systematic bias under minor perturbations of the validation set—a phenomenon that challenges the common assumption of their robustness. The study is the first to identify and rigorously characterize how structural changes in the validation set, such as the introduction of label noise, induce a compression of Shapley values toward zero. Leveraging the KNN-Shapley theoretical framework and supported by experiments on both synthetic and real-world datasets, the authors elucidate the underlying mechanisms and patterns of this bias. To mitigate its effects, they propose normalization and boundary-aware validation strategies that substantially enhance the robustness, stability, and interpretability of data valuation, offering a principled foundation for reliable data pricing in machine learning markets.

Abstract

Shapley values are widely used to attribute value to training data based on their marginal contribution to performance on a validation set. Existing practice often assumes these values are stable once the training data and model are fixed. In this work, we uncover a systematic vulnerability: even modest changes to the validation set, such as introducing noises, cause directional shifts in Shapley distributions. As noises are added, Shapley values of training samples compress toward zero. We trace this to a noise-induced neighborhood reshuffling effect: perturbations alter the local rank order between validation and training samples, flattening the valuation landscape. Using the KNN-Shapley framework, we show through synthetic and real data that these shifts are consistent and reproducible. Our findings challenge the assumption of Shapley stability and reveal a new axis of fragility in data valuation. We propose normalization and boundary-aware validation strategies to mitigate these distortions and enable more robust, interpretable valuation in machine learning marketplaces.

Problem

• Shapley values
• data valuation
• validation set
• noise robustness
• valuation stability

Innovation

• Shapley values
• data valuation
• validation set perturbation
• neighborhood reshuffling
• KNN-Shapley