CoCoAFusE: Beyond Mixtures of Experts via Model Fusion

2025-05-02

AI Summary

This paper addresses the dual challenges of poor interpretability and low uncertainty quantification (UQ) accuracy in deep learning for multimodal tasks with covariate-dependent uncertainty. We propose the Competitive/Cooperative Adaptive Fusion of Experts (CoCoAFusE), a novel mixture-of-experts (MoE) framework that introduces *distribution-level fusion*—replacing conventional weighted averaging with cooperative probabilistic modeling across expert outputs. This enables locally interpretable, mechanism-aware uncertainty estimation. A key innovation is the elimination of multimodal artifacts in smooth transition regions, thereby improving prediction interval tightness and calibration. Extensive experiments on multiple synthetic and real-world benchmarks demonstrate that CoCoAFusE significantly enhances UQ performance: it yields narrower, better-calibrated prediction intervals while maintaining high predictive accuracy and preserving local interpretability grounded in underlying data mechanisms.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Reasoning under Uncertainty: Uncertainty Representations
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Many learning problems involve multiple patterns and varying degrees of uncertainty dependent on the covariates. Advances in Deep Learning (DL) have addressed these issues by learning highly nonlinear input-output dependencies. However, model interpretability and Uncertainty Quantification (UQ) have often straggled behind. In this context, we introduce the Competitive/Collaborative Fusion of Experts (CoCoAFusE), a novel, Bayesian Covariates-Dependent Modeling technique. CoCoAFusE builds on the very philosophy behind Mixtures of Experts (MoEs), blending predictions from several simple sub-models (or"experts") to achieve high levels of expressiveness while retaining a substantial degree of local interpretability. Our formulation extends that of a classical Mixture of Experts by contemplating the fusion of the experts' distributions in addition to their more usual mixing (i.e., superimposition). Through this additional feature, CoCoAFusE better accommodates different scenarios for the intermediate behavior between generating mechanisms, resulting in tighter credible bounds on the response variable. Indeed, only resorting to mixing, as in classical MoEs, may lead to multimodality artifacts, especially over smooth transitions. Instead, CoCoAFusE can avoid these artifacts even under the same structure and priors for the experts, leading to greater expressiveness and flexibility in modeling. This new approach is showcased extensively on a suite of motivating numerical examples and a collection of real-data ones, demonstrating its efficacy in tackling complex regression problems where uncertainty is a key quantity of interest.

Problem

• Enhances model interpretability and uncertainty quantification in deep learning
• Extends Mixtures of Experts by fusing expert distributions for better flexibility
• Addresses multimodality artifacts in smooth transitions between generating mechanisms

Innovation

• Bayesian Covariates-Dependent Modeling technique
• Competitive/Collaborative Fusion of Experts
• Fusion of experts' distributions for flexibility