MixFormer: Co-Scaling Up Dense and Sequence in Industrial Recommenders

2026-02-15

AI Summary

This work addresses the limitation in industrial recommendation systems where decoupling sequential modeling and feature interaction modules hinders synergistic scaling under constrained computational budgets. To overcome this, we propose MixFormer—a unified Transformer architecture that jointly models user behavioral sequences and high-order feature interactions within a single backbone, for the first time integrating both components into a shared parameterized framework. By introducing a user-item decoupling strategy, MixFormer significantly enhances inference efficiency and deployment feasibility. Extensive evaluations on large-scale industrial datasets and online A/B tests on Douyin and Douyin Lite demonstrate consistent and significant improvements across key metrics, including recommendation accuracy, user active days, and session duration.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Data Mining & Knowledge Management: Recommender Systems
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

As industrial recommender systems enter a scaling-driven regime, Transformer architectures have become increasingly attractive for scaling models towards larger capacity and longer sequence. However, existing Transformer-based recommendation models remain structurally fragmented, where sequence modeling and feature interaction are implemented as separate modules with independent parameterization. Such designs introduce a fundamental co-scaling challenge, as model capacity must be suboptimally allocated between dense feature interaction and sequence modeling under a limited computational budget. In this work, we propose MixFormer, a unified Transformer-style architecture tailored for recommender systems, which jointly models sequential behaviors and feature interactions within a single backbone. Through a unified parameterization, MixFormer enables effective co-scaling across both dense capacity and sequence length, mitigating the trade-off observed in decoupled designs. Moreover, the integrated architecture facilitates deep interaction between sequential and non-sequential representations, allowing high-order feature semantics to directly inform sequence aggregation and enhancing overall expressiveness. To ensure industrial practicality, we further introduce a user-item decoupling strategy for efficiency optimizations that significantly reduce redundant computation and inference latency. Extensive experiments on large-scale industrial datasets demonstrate that MixFormer consistently exhibits superior accuracy and efficiency. Furthermore, large-scale online A/B tests on two production recommender systems, Douyin and Douyin Lite, show consistent improvements in user engagement metrics, including active days and in-app usage duration.

Problem

• co-scaling
• Transformer-based recommendation
• sequence modeling
• feature interaction
• industrial recommender systems

Innovation

• MixFormer
• co-scaling
• unified architecture
• feature interaction
• sequence modeling