User Profile with Large Language Models: Construction, Updating, and Benchmarking

2025-02-15

AI Summary

To address the limited accuracy and weak dynamic update capability in user profiling, this paper proposes a probabilistic large language model (LLM)-driven framework for dynamic profile construction and incremental updating. Methodologically, we introduce a dual-task benchmark: (i) the first high-quality open-source dataset tailored for dynamic profiling scenarios—comprising distinct subsets for profile construction and incremental update—and (ii) a context-aware, interpretable profiling approach leveraging Mistral-7B and Llama2-7B, integrating probabilistic modeling with structured information extraction. Experimental results demonstrate substantial improvements in precision and recall across multiple evaluation dimensions, achieving state-of-the-art (SOTA) performance. The framework validates the effectiveness, generalizability, and practical utility of LLMs in user profiling tasks, particularly under evolving user behaviors and sparse or noisy data conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

User profile modeling plays a key role in personalized systems, as it requires building accurate profiles and updating them with new information. In this paper, we present two high-quality open-source user profile datasets: one for profile construction and another for profile updating. These datasets offer a strong basis for evaluating user profile modeling techniques in dynamic settings. We also show a methodology that uses large language models (LLMs) to tackle both profile construction and updating. Our method uses a probabilistic framework to predict user profiles from input text, allowing for precise and context-aware profile generation. Our experiments demonstrate that models like Mistral-7b and Llama2-7b perform strongly in both tasks. LLMs improve the precision and recall of the generated profiles, and high evaluation scores confirm the effectiveness of our approach.

Problem

• User profile modeling
• Large language models
• Dynamic settings

Innovation

• Open-source datasets for profiles
• LLMs for profile construction
• Probabilistic framework for prediction