CMSL: Constructive Multi-Sequence Learning for Recommendation Systems

2026-06-26

AI Summary

This work addresses the limitations of traditional sequential recommendation models, which represent user behavior as a single sequence and consequently suffer from mixed multi-interest signals and contextual contamination, hindering the capture of high-intent actions. To overcome this, the authors propose a constructive multi-sequence learning framework that employs a learnable sequence construction module to explicitly disentangle user history in latent space, generating thematically coherent subsequences. Coupled with a linear attention mechanism, this approach enables efficient and focused modeling. Departing from the conventional single-sequence paradigm, the method introduces the concept of “context engineering” to achieve clean, disentangled representations of multiple user interests. The framework has been deployed across four core recommendation scenarios at Meta, demonstrating significant improvements in recommendation accuracy and model focus for both ranking and retrieval tasks.

Abstract

Sequence learning has emerged as the promising paradigm in recommendation systems, surpassing traditional Deep Learning Recommendation Models (DLRM) by capturing the temporal nuances of user behavior. However, current state-of-the-art architectures operate under a limiting analogy: they treat user history as a monolithic chronological sequence like a sentence in a Large Language Model (LLM). We observe a fundamental divergence between natural language and recommendation data: unlike the linear, logical flow of text, user history is inherently multi-faceted. A user's journey is a fragmented reflection of diverse interests, resulting in much weaker coherence between items than is found in LLM training data. This lack of structural unity leads to context pollution. In single-sequence modeling, unrelated behaviors compete for the same attention budget. This "noisy" signal dilutes the model's focus, effectively capping its ability to discern high-intent patterns from background activity. To address this, we propose Constructive Multi-Sequence Learning (CMSL), a paradigm shift from passive sequence ingestion to active "context engineering" that constructs multiple coherent sequences in latent space. CMSL leverages a learnable Sequence Construction Module to disentangle user history into "pure" thematic strands, followed by a linear attention mechanism to efficiently model these strands at scale. CMSL has been deployed across ranking and retrieval tasks and across four major surfaces at Meta.

Problem

• sequence learning
• recommendation systems
• context pollution
• user behavior modeling
• multi-faceted interests

Innovation

• Multi-Sequence Learning
• Context Engineering
• Sequence Disentanglement
• Linear Attention
• Recommendation Systems