Varying-Coefficient Mixture of Experts Model

2026-01-05

AI Summary

This work proposes a varying-coefficient mixture-of-experts model (VCMoE) to address the limitation of traditional mixture-of-experts frameworks, which assume constant covariate effects and thus struggle to capture heterogeneity in dynamic settings. By introducing varying-coefficient modeling into this architecture for the first time, VCMoE allows both expert and gating network coefficients to evolve smoothly with an index variable, flexibly characterizing the dynamic nature of covariate effects. The authors establish theoretical guarantees regarding model identifiability, estimation consistency, and inference, develop a label-consistent EM algorithm, and construct simultaneous confidence bands via asymptotic theory and bootstrap methods. A generalized likelihood ratio test is devised to assess whether coefficients genuinely vary. Simulations confirm accurate estimation and proper coverage, while application to single-nucleus gene expression data from mouse embryonic brains reveals temporally varying regulatory dynamics of Satb2 and Bcl11b across two neuronal subtypes, aligning with established biological mechanisms.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Mixture-of-Experts (MoE) is a flexible framework that combines multiple specialized submodels (``experts''), by assigning covariate-dependent weights (``gating functions'') to each expert, and have been commonly used for analyzing heterogeneous data. Existing statistical MoE formulations typically assume constant coefficients, for covariate effects within the expert or gating models, which can be inadequate for longitudinal, spatial, or other dynamic settings where covariate influences and latent subpopulation structure evolve across a known dimension. We propose a Varying-Coefficient Mixture of Experts (VCMoE) model that allows all coefficient effects in both the gating functions and expert models to vary along an indexing variable. We establish identifiability and consistency of the proposed model, and develop an estimation procedure, label-consistent EM algorithm, for both fully functional and hybrid specifications, along with the corresponding asymptotic distributions of the resulting estimators. For inference, simultaneous confidence bands are constructed using both asymptotic theory for the maximum discrepancy between the estimated functional coefficients and their true counterparts, and with bootstrap methods. In addition, a generalized likelihood ratio test is developed to examine whether a coefficient function is genuinely varying across the index variable. Simulation studies demonstrate good finite-sample performance, with acceptable bias and satisfactory coverage rates. We illustrate the proposed VCMoE model using a dataset of single nucleus gene expression in embryonic mice to characterize the temporal dynamics of the associations between the expression levels of genes Satb2 and Bcl11b across two latent cell subpopulations of neurons, yielding results that are consistent with prior findings.

Problem

• Mixture-of-Experts
• varying-coefficient
• heterogeneous data
• dynamic settings
• latent subpopulation

Innovation

• Varying-Coefficient Model
• Mixture of Experts
• Dynamic Heterogeneity
• Label-Consistent EM Algorithm
• Generalized Likelihood Ratio Test