Revisiting Data Attribution for Influence Functions

2025-08-10

AI Summary

This work addresses the problem of data attribution in deep learning—specifically, identifying the training samples most influential to a given model prediction. We propose a scalable influence-function-based attribution framework that leverages first-order Taylor approximations and robust statistical theory to estimate, without retraining, the gradient-level impact of individual training examples on both model parameters and predictions. We systematically survey and enhance efficient algorithms for computing the inverse-Hessian-vector product (HVP), significantly improving their practicality for large-scale models and datasets. Experiments demonstrate high accuracy and interpretability of our method in applications including mislabeled example detection and critical sample identification. The approach provides both theoretical foundations and practical implementation pathways for enhancing the trustworthiness of deep learning systems.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The goal of data attribution is to trace the model's predictions through the learning algorithm and back to its training data. thereby identifying the most influential training samples and understanding how the model's behavior leads to particular predictions. Understanding how individual training examples influence a model's predictions is fundamental for machine learning interpretability, data debugging, and model accountability. Influence functions, originating from robust statistics, offer an efficient, first-order approximation to estimate the impact of marginally upweighting or removing a data point on a model's learned parameters and its subsequent predictions, without the need for expensive retraining. This paper comprehensively reviews the data attribution capability of influence functions in deep learning. We discuss their theoretical foundations, recent algorithmic advances for efficient inverse-Hessian-vector product estimation, and evaluate their effectiveness for data attribution and mislabel detection. Finally, highlighting current challenges and promising directions for unleashing the huge potential of influence functions in large-scale, real-world deep learning scenarios.

Problem

• Trace model predictions to influential training data
• Understand impact of training samples on predictions
• Improve interpretability and accountability in machine learning

Innovation

• Influence functions trace predictions to training data
• Efficient inverse-Hessian-vector product estimation
• Data attribution and mislabel detection applications