Tuning Language Models for Robust Prediction of Diverse User Behaviors

2025-05-23

AI Summary

Existing user behavior prediction methods struggle to jointly model high-frequency “anchor” behaviors and low-frequency “tail” behaviors, exhibiting insufficient generalization in long-tail scenarios. This paper proposes BehaviorLM, a two-stage progressive fine-tuning framework leveraging large language models (LLMs). In the first stage, pretrained general behavioral knowledge is preserved; in the second stage, sample difficulty estimation and anchor-tail stratified sampling are introduced to specifically enhance tail-behavior modeling. To our knowledge, this is the first work to integrate a difficulty-aware balanced training paradigm into LLM-based behavioral fine-tuning, overcoming the traditional fine-tuning bias against long-tail behaviors. Evaluated on two real-world datasets, BehaviorLM improves tail-behavior F1-score by 18.7% while preserving anchor-behavior performance, and achieves effective tail-behavior modeling with as few as five samples.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Behavior Learning & Control

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Predicting user behavior is essential for intelligent assistant services, yet deep learning models often struggle to capture long-tailed behaviors. Large language models (LLMs), with their pretraining on vast corpora containing rich behavioral knowledge, offer promise. However, existing fine-tuning approaches tend to overfit to frequent ``anchor'' behaviors, reducing their ability to predict less common ``tail'' behaviors. In this paper, we introduce BehaviorLM, a progressive fine-tuning approach that addresses this issue. In the first stage, LLMs are fine-tuned on anchor behaviors while preserving general behavioral knowledge. In the second stage, fine-tuning uses a balanced subset of all behaviors based on sample difficulty to improve tail behavior predictions without sacrificing anchor performance. Experimental results on two real-world datasets demonstrate that BehaviorLM robustly predicts both anchor and tail behaviors and effectively leverages LLM behavioral knowledge to master tail behavior prediction with few-shot examples.

Problem

• Predicting diverse user behaviors robustly using LLMs
• Overfitting to frequent behaviors reduces tail behavior prediction
• Balancing sample difficulty improves tail behavior prediction accuracy

Innovation

• Progressive fine-tuning for diverse behaviors
• Balanced subset training by sample difficulty
• Few-shot learning for tail behavior prediction