Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs

2026-04-14

AI Summary

Existing attribution methods are primarily designed for encoder architectures and struggle to effectively capture the causal and semantic complexities inherent in decoder-only autoregressive language models. This work proposes HETA, a novel framework that, for the first time, incorporates second-order Hessian information into token attribution. By integrating semantic shift vectors, Hessian sensitivity scores, and KL divergence masking, HETA establishes the first unified attribution approach tailored for generative settings—offering context awareness, causal fidelity, and semantic plausibility. Accompanied by a dedicated evaluation benchmark, comprehensive experiments demonstrate that HETA significantly outperforms current methods across multiple mainstream models and datasets, achieving new state-of-the-art performance in both attribution faithfulness and alignment with human annotations.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose Hessian-Enhanced Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.

Problem

• attribution
• autoregressive language models
• decoder-only models
• interpretability
• token contribution

Innovation

• Hessian-based attribution
• autoregressive language models
• token attribution
• semantic transition vector
• interpretability