Shift is Good: Mismatched Data Mixing Improves Test Performance

2025-10-28

AI Summary

This paper investigates distribution shift arising from mismatched training-to-test proportions across subpopulations—i.e., when the mixture proportions differ between training and test distributions—even in the absence of statistical dependencies or transferable structures among subpopulations. Method: We formalize the problem via mixture distribution modeling, conduct rigorous theoretical analysis, and derive closed-form solutions for the optimal training mixture proportions that maximize test performance. Contribution/Results: We identify and characterize the counterintuitive phenomenon of “beneficial distribution shift”: deliberately deviating from proportional sampling can significantly improve generalization. We establish tight bounds on the achievable performance gain and derive the optimal training proportions under diverse multi-scenario settings. Furthermore, we extend our framework to practical applications such as skill composition tasks. This work broadens the distribution shift research paradigm by providing interpretable, theoretically grounded principles for data proportion design—offering both conceptual insight and actionable guidelines for real-world deployment.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We consider training and testing on mixture distributions with different training and test proportions. We show that in many settings, and in some sense generically, distribution shift can be beneficial, and test performance can improve due to mismatched training proportions, even if the components are unrelated and with no transfer between components. In a variety of scenarios, we identify the optimal training proportions and the extent to which such distribution shift can be beneficial. We show how the same analysis applies also to a compositional setting with differing distribution of component "skills'' at training and test.

Problem

• Investigates beneficial effects of distribution shift on test performance
• Identifies optimal training proportions for mismatched data mixtures
• Extends analysis to compositional settings with varying skill distributions

Innovation

• Mismatched data mixing improves test performance
• Optimal training proportions identified for distribution shift
• Analysis applies to compositional skill distribution settings