Approximations to worst-case data dropping: unmasking failure modes

2024-08-16

AI Summary

This paper addresses the challenge of robustness diagnostics in ordinary least squares (OLS) linear regression—specifically, whether removing a small number of data points induces substantial changes in statistical inference. We demonstrate that mainstream approximation methods systematically fail to detect influential subsets under realistic data configurations. To resolve this, we propose a recursive greedy algorithm and provide the first rigorous proof of its 100% detection success rate across multiple benchmark datasets. Comprehensive evaluations on both synthetic and real-world data show that our method achieves perfect detection accuracy (zero failure rate), outperforming existing approaches by up to several orders of magnitude in computational efficiency. Our core contribution lies in exposing the fundamental theoretical limitations of approximate influence assessment and delivering the first exact solution that simultaneously offers provable guarantees and practical reliability.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

A data analyst might worry about generalization if dropping a very small fraction of data points from a study could change its substantive conclusions. Checking this non-robustness directly poses a combinatorial optimization problem and is intractable even for simple models and moderate data sizes. Recently various authors have proposed a diverse set of approximations to detect this non-robustness. In the present work, we show that, even in a setting as simple as ordinary least squares (OLS) linear regression, many of these approximations can fail to detect (true) non-robustness in realistic data arrangements. We focus on OLS in the present work due its widespread use and since some approximations work only for OLS. Across our synthetic and real-world data sets, we find that a simple recursive greedy algorithm is the sole algorithm that does not fail any of our tests and also that it can be orders of magnitude faster to run than some competitors.

Problem

• Detecting non-robustness in data analysis conclusions
• Evaluating approximations for worst-case data dropping
• Assessing algorithm performance in OLS linear regression

Innovation

• Recursive greedy algorithm detects non-robustness
• Approximations fail in OLS linear regression
• Efficient solution for worst-case data dropping