LUMOS: Large User MOdels for User Behavior Prediction

2025-11-28

AI Summary

To address poor generalizability and scalability in large-scale user behavior prediction on online B2C platforms—caused by heavy reliance on manual feature engineering, task-specific models, and domain expertise—this paper proposes the first unified large model for massive-user behavioral forecasting. The model operates solely on raw user behavioral sequences, employing multimodal tokenization to jointly encode temporal actions, event contexts, and static demographic attributes. It introduces a novel conditional cross-attention mechanism explicitly designed for future events (e.g., promotions, holidays), enabling causal reasoning. Built upon the Transformer architecture, it adopts end-to-end joint multi-task learning. Evaluated on real-world data comprising 275 billion tokens and 250 million users, the model achieves an average ROC-AUC improvement of 0.025 and a 4.6% reduction in MAPE across five prediction tasks. Online A/B testing demonstrates a 3.15% increase in daily active users.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

User behavior prediction at scale remains a critical challenge for online B2C platforms. Traditional approaches rely heavily on task-specific models and domain-specific feature engineering. This is time-consuming, computationally expensive, and requires domain expertise and therefore not scalable. We present LUMOS (Large User MOdel Series), a transformer-based architecture that eliminates task-specific models and manual feature engineering by learning multiple tasks jointly using only raw user activity data. LUMOS introduces a novel cross-attention mechanism that conditions predictions on future known events (e.g., holidays, sales, etc.), enabling the model to predict complex behaviour patterns like"how will upcoming holidays affect user engagement?"The architecture also employs multi-modal tokenization, combining user transactions, event context, and static user demographic attributes into rich representations processed through specialized embedding pathways. Through extensive experiments on a production dataset spanning 275 billion user activity tokens from 250 million users, we demonstrate that LUMOS achieves superior performance compared to traditional task-specific models. Across 5 tasks with established baselines, we achieve an average improvement of 0.025 in ROC-AUC for binary classification tasks and 4.6% reduction in MAPE for regression tasks. Online A/B testing validates these improvements translate to measurable business impact with a 3.15% increase in Daily Active Users.

Problem

• Predicts user behavior at scale without task-specific models
• Eliminates manual feature engineering using raw activity data
• Incorporates future known events to forecast complex behavior patterns

Innovation

• Transformer-based architecture for joint multi-task learning
• Cross-attention mechanism conditioning on future known events
• Multi-modal tokenization combining user activities and attributes