LLMs as Better Recommenders with Natural Language Collaborative Signals: A Self-Assessing Retrieval Approach

2025-05-26

AI Summary

To address semantic misalignment caused by conventional collaborative information (CI) encoding in LLM-based recommendation, this paper proposes a natural-language collaborative signal modeling approach: user behavioral sequences are converted into interpretable textual descriptions, and a two-stage self-assessment retrieval framework—Collaborative-Aware Retrieval (CAR) followed by Self-Adaptive Re-ranking via Reasoning (SARE)—is introduced. CAR first performs coarse-grained retrieval of highly relevant behaviors, and SARE leverages LLMs’ intrinsic reasoning capabilities for fine-grained re-ranking. Departing from traditional soft tokenization or ID-based embeddings, our method bridges the semantic gap by enabling native alignment between collaborative patterns and language models. Experiments on two public benchmarks demonstrate significant improvements over state-of-the-art methods in Recall@10 and NDCG@10, validating the effectiveness, interpretability, and generalizability of natural-language collaborative signals.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Incorporating collaborative information (CI) effectively is crucial for leveraging LLMs in recommendation tasks. Existing approaches often encode CI using soft tokens or abstract identifiers, which introduces a semantic misalignment with the LLM's natural language pretraining and hampers knowledge integration. To address this, we propose expressing CI directly in natural language to better align with LLMs' semantic space. We achieve this by retrieving a curated set of the most relevant user behaviors in natural language form. However, identifying informative CI is challenging due to the complexity of similarity and utility assessment. To tackle this, we introduce a Self-assessing COllaborative REtrieval framework (SCORE) following the retrieve-rerank paradigm. First, a Collaborative Retriever (CAR) is developed to consider both collaborative patterns and semantic similarity. Then, a Self-assessing Reranker (SARE) leverages LLMs' own reasoning to assess and prioritize retrieved behaviors. Finally, the selected behaviors are prepended to the LLM prompt as natural-language CI to guide recommendation. Extensive experiments on two public datasets validate the effectiveness of SCORE in improving LLM-based recommendation.

Problem

• Encode collaborative information in natural language for LLMs
• Retrieve relevant user behaviors with semantic alignment
• Assess and prioritize behaviors using LLM self-reasoning

Innovation

• Expressing collaborative information in natural language
• Self-assessing retrieval framework for informative behaviors
• Retrieve-rerank approach with LLM reasoning