Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time

2026-09-04

AI Summary

提出一种影响评分方法，量化注意力头在基于Transformer的模型中对分类决策的贡献，用于提升模型在提示注入检测中的解释性。

Abstract

We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on the logits with structural contribution within the residual stream, enabling a multi-scale analysis at the head, layer, and network levels. Applied to a DeBERTa model specialized for prompt injection detection, our framework reveals distinct decision behaviours between correct and erroneous predictions. Our method provides an effective compromise between fine-grained circuit analysis and global output-based methods, and offers a systematic way to study decision mechanisms in Transformer classifiers.

Problem

• influence score
• attention heads
• classification decisions
• prompt injection detection
• transformers

Innovation

• influence score
• attention heads
• multi-scale analysis
• prompt injection detection
• DeBERTa