TenExp: Mixture-of-Experts-Based Tensor Decomposition Structure Search Framework

2026-03-03

AI Summary

Automatically selecting or combining appropriate tensor decomposition structures to accurately capture the low-rank properties inherent in data remains a significant challenge in the field. This work proposes an unsupervised tensor decomposition architecture search framework based on a Mixture-of-Experts mechanism, which for the first time enables dynamic and adaptive switching between single and hybrid decomposition structures. This approach overcomes the limitation of conventional methods that are confined to fixed families of factor interactions. Theoretically, the method provides approximation error bounds, and empirically, it demonstrates substantial improvements over state-of-the-art approaches on both synthetic and real-world datasets, thereby validating its effectiveness and generalization capability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Recently, tensor decompositions continue to emerge and receive increasing attention. Selecting a suitable tensor decomposition to exactly capture the low-rank structures behind the data is at the heart of the tensor decomposition field, which remains a challenging and relatively under-explored problem. Current tensor decomposition structure search methods are still confined by a fixed factor-interaction family (e.g., tensor contraction) and cannot deliver the mixture of decompositions. To address this problem, we elaborately design a mixture-of-experts-based tensor decomposition structure search framework (termed as TenExp), which allows us to dynamically select and activate suitable tensor decompositions in an unsupervised fashion. This framework enjoys two unique advantages over the state-of-the-art tensor decomposition structure search methods. Firstly, TenExp can provide a suitable single decomposition beyond a fixed factor-interaction family. Secondly, TenExp can deliver a suitable mixture of decompositions beyond a single decomposition. Theoretically, we also provide the approximation error bound of TenExp, which reveals the approximation capability of TenExp. Extensive experiments on both synthetic and realistic datasets demonstrate the superiority of the proposed TenExp compared to the state-of-the-art tensor decomposition-based methods.

Problem

• tensor decomposition
• structure search
• low-rank structure
• mixture of decompositions
• factor-interaction family

Innovation

• Mixture-of-Experts
• Tensor Decomposition
• Structure Search
• Unsupervised Learning
• Low-rank Approximation