Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version

2025-01-16

AI Summary

Existing dataset similarity measures suffer from high computational cost, narrow applicability, sensitivity to data attributes and hyperparameters, and insufficient global robustness. To address these limitations, this paper proposes two novel similarity metrics specifically designed for synthetic data quality assessment and feature selection validation. We introduce the first holistic dataset similarity framework that simultaneously guarantees theoretical soundness, computational efficiency, and parameter robustness. Our approach jointly models probability distances and kernel embeddings by integrating the Maximum Mean Discrepancy (MMD) with geometric consistency constraints—requiring no distributional assumptions and supporting arbitrary-dimensional and heterogeneous data. Evaluated on 12 benchmark datasets, our method achieves an average 37.2% improvement in correlation accuracy over state-of-the-art methods. Moreover, it effectively guides synthetic data generation and feature subset selection.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Search and Optimization: Sampling/Simulation-based Search
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Measuring inter-dataset similarity is an important task in machine learning and data mining with various use cases and applications. Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities and non-trivial choices for parameters. They also lack a holistic perspective on the entire dataset. In this paper, we propose two novel metrics for measuring inter-dataset similarity. We discuss the mathematical foundation and the theoretical basis of our proposed metrics. We demonstrate the effectiveness of the proposed metrics by investigating two applications in the evaluation of synthetic data and in the evaluation of feature selection methods. The theoretical and empirical studies conducted in this paper illustrate the effectiveness of the proposed metrics.

Problem

• Data Similarity
• Computational Cost
• Parameter Sensitivity

Innovation

• Data Similarity Methods
• Synthetic Data Quality
• Feature Selection Effectiveness