Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders

2026-08-14

AI Summary

This study addresses the unexplained over-reliance on recent interactions during inference in causal self-attentive recommendation models by proposing the concept of "residual dominance" and establishing a structured explanatory framework for last-item dependency. Through norm analysis, we demonstrate a monotonic correlation between residual magnitude and last-item dependency, subsequently designing an inference-time residual scaling intervention to balance information mixing with locality. Experimental results indicate that this intervention effectively mitigates the residual dominance effect, rectifies misclassified samples, and significantly improves recommendation accuracy. Consequently, this work provides novel diagnostic perspectives and theoretical foundations for understanding and optimizing sequential recommendation models, offering a principled approach to enhancing their interpretability and performance during the inference phase.

Abstract

Transformer-based sequential recommenders with causal self-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for prediction remains unclear. We combine prediction-time diagnostics with norm-based analysis of the full attention block. First, we show that SASRec-style models exhibit highly localized last-item reliance. We then find that, although self-attention aggregates contextual information, residual addition sharply shifts the full-block representation toward same-position contributions, which we term residual dominance. To probe this interpretation, we use inference-time residual scaling as a controlled diagnostic intervention. Changing the residual strength induces a monotonic trade-off between structural mixing and last-item reliance, while reducing residual strength recovers a subset of final-position misses for which representations at non-final positions already rank the ground-truth item correctly. Our results provide a structural account linking extreme last-item reliance to residual dominance at inference time. The code is publicly available.

Problem

• Causal Self-Attention
• Last-Item Reliance
• Residual Dominance
• Sequential Recommenders

Innovation

• Residual Dominance
• Last-Item Reliance
• Causal Self-Attention
• Sequential Recommendation
• Residual Scaling