Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions

2025-09-27

AI Summary

This work investigates how the fidelity of Hessian matrix approximations—such as the Generalized Gauss–Newton (GGN), Kronecker-Factored Approximate Curvature (K-FAC), and Eigenvalue-Corrected K-FAC (EK-FAC)—affects influence-function-based data attribution in deep learning, a critical yet under-studied issue due to the Hessian’s inherent ill-conditioning. We quantitatively evaluate the accuracy of various curvature approximations and their resulting attribution quality on classification tasks. Our analysis reveals that higher-fidelity Hessian approximations consistently improve the quality of influence scores; notably, eigenvalue mismatch between K-FAC and GGN/EK-FAC emerges as the dominant source of attribution error. Through ablation across approximation steps, we provide the first systematic decomposition of each component’s contribution to attribution performance. Results confirm that improved Hessian approximation significantly enhances attribution accuracy. Moreover, our findings yield interpretable, principled guidance for trading off computational efficiency against attribution fidelity in influence-function applications.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Influence functions offer a principled way to trace model predictions back to training data, but their use in deep learning is hampered by the need to invert a large, ill-conditioned Hessian matrix. Approximations such as Generalised Gauss-Newton (GGN) and Kronecker-Factored Approximate Curvature (K-FAC) have been proposed to make influence computation tractable, yet it remains unclear how the departure from exactness impacts data attribution performance. Critically, given the restricted regime in which influence functions are derived, it is not necessarily clear better Hessian approximations should even lead to better data attribution performance. In this paper, we investigate the effect of Hessian approximation quality on influence-function attributions in a controlled classification setting. Our experiments show that better Hessian approximations consistently yield better influence score quality, offering justification for recent research efforts towards that end. We further decompose the approximation steps for recent Hessian approximation methods and evaluate each step's influence on attribution accuracy. Notably, the mismatch between K-FAC eigenvalues and GGN/EK-FAC eigenvalues accounts for the majority of the error and influence loss. These findings highlight which approximations are most critical, guiding future efforts to balance computational tractability and attribution accuracy.

Problem

• Evaluating how Hessian approximations affect influence function data attribution accuracy
• Comparing GGN and K-FAC curvature approximations' impact on influence scores
• Identifying which approximation steps most degrade influence attribution performance

Innovation

• Better Hessian approximations improve influence score quality
• Decomposing approximation steps reveals key error sources
• K-FAC eigenvalue mismatch causes major influence attribution loss