HatLLM: Hierarchical Attention Masking for Enhanced Collaborative Modeling in LLM-based Recommendation

2025-10-12

AI Summary

Existing LLM-based sequential recommendation methods struggle to effectively model collaborative signals among user behaviors, primarily because standard attention mechanisms over-emphasize intra-item token interactions while weakening cross-item behavioral associations. To address this, we propose Hierarchical Attention Masking (HAM), a novel attention mechanism that decouples and jointly models semantic understanding and collaborative signal learning: at shallow layers, it constrains token-level semantic modeling via structural masking; at deeper layers, it explicitly guides item-level collaborative relationship learning through hierarchical attention masks. Our approach integrates structured behavioral sequence inputs, a hierarchically designed masking strategy, and lightweight LLM fine-tuning. Extensive experiments on three real-world datasets demonstrate an average improvement of 9.13% over state-of-the-art LLM-based recommenders. This work is the first to systematically resolve the attention allocation imbalance in LLMs that undermines collaborative modeling—establishing a principled framework for effective sequential recommendation with large language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent years have witnessed a surge of research on leveraging large language models (LLMs) for sequential recommendation. LLMs have demonstrated remarkable potential in inferring users' nuanced preferences through fine-grained semantic reasoning. However, they also exhibit a notable limitation in effectively modeling collaborative signals, i.e., behavioral correlations inherent in users' historical interactions. Our empirical analysis further reveals that the attention mechanisms in LLMs tend to disproportionately focus on tokens within the same item, thereby impeding the capture of cross-item correlations. To address this limitation, we propose a novel hierarchical attention masking strategy for LLM-based recommendation, termed HatLLM. Specifically, in shallow layers, HatLLM masks attention between tokens from different items, facilitating intra-item semantic understanding; in contrast, in deep layers, HatLLM masks attention within items, thereby compelling the model to capture cross-item correlations. This progressive, layer-wise approach enables LLMs to jointly model both token-level and item-level dependencies. Extensive experiments on three real-world datasets demonstrate that HatLLM achieves significant performance gains (9.13% on average) over existing LLM-based methods.

Problem

• Addresses LLMs' limitation in modeling collaborative recommendation signals
• Solves disproportionate attention focus on same-item tokens in LLMs
• Enables joint modeling of token-level and item-level dependencies

Innovation

• Hierarchical attention masking strategy for LLM recommendation
• Shallow layers mask attention between different items
• Deep layers mask attention within items for correlations