How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines

2026-05-12

AI Summary

Existing influence estimation methods for trajectory-based data attribution lack systematic error analysis, limiting their reliability in data selection, valuation, and model diagnostics. This work identifies optimizer mismatch—specifically with AdamW—as a critical source of error and introduces AdamW-influence to address it. We derive a closed-form proxy for influence that avoids costly retraining and propose a K-step lookahead framework that unifies offline and online data selection strategies. Experiments across MLPs, CNNs, GPT-2, and Llama 3.2-1B demonstrate that our approach improves attribution accuracy by 10% to 300%, with online short-horizon selection matching or even surpassing the performance of offline methods.

Abstract

Trajectory-based data attribution methods estimate the influence of training samples on model predictions by unrolling the training trajectory. They are widely used in applications such as data selection, data valuation, and model diagnosis, but there is a lack of comprehensive error analysis of these methods, raising concerns about method faithfulness and hindering reliable deployment. In this work, we provide the first systematic analysis of error sources in trajectory-based data attribution, together with concrete remedies to mitigate them and practical guidelines for downstream use. We organize the total error into three categories, config-level, algorithm-level, and system-level. We make three contributions. First, we identify optimizer mismatch as the dominant config-level error: existing methods derive their attribution under the assumption of SGD, even for models trained with the modern de facto optimizer AdamW. We propose AdamW-influence to fully account for AdamW's optimization dynamics, yielding improvements from 10% to over 300% in Spearman correlation between estimated and ground-truth influence across four settings spanning MLP, CNN, GPT-2, and Llama 3.2-1B. Second, we isolate the remaining algorithm-level error arising from the first-order Taylor approximation, identify the learning rate and trajectory length as factors governing the error magnitude, and derive a closed-form error proxy that can be evaluated along the original trajectory without retraining. Third, we translate these insights into practical guidelines for data selection by unifying offline and online strategies under a K-step look-ahead framework. Under this framework, online selection with a short horizon often matches or exceeds offline, and the optimal horizon can be tuned jointly with the learning rate. Together, these results turn the framework into an actionable selection recipe for practitioners.

Problem

• data attribution
• trajectory-based methods
• error analysis
• faithfulness
• influence estimation

Innovation

• trajectory-based attribution
• AdamW-influence
• error analysis
• data selection
• influence estimation