modality reliability estimation

This work addresses the unclear role of reliability scores in existing quality-aware multimodal fusion methods—specifically, whether these scores genuinely guide model decisions. To investigate this, the authors propose a leakage-safe diagnostic approach: during inference, the model is frozen and reliability scores are shuffled across test samples to assess their actual impact on final predictions. This method effectively distinguishes whether the scores actively drive fusion decisions or merely correlate with performance. Experiments on the StressID and CMU-MOSEI datasets reveal that shuffling scores has negligible effect on performance in real-world scenarios; significant gains from the fusion mechanism occur only when the reliability scores accurately predict the correctness of individual modalities.

This work addresses the challenge that modality reliability in multimodal fusion often varies dynamically with context—such as sensor degradation or class-specific noise—rendering traditional methods based on static assumptions ineffective. To overcome this limitation, the authors propose the C²MF framework, which introduces a Context-Specific Information Credibility (CSIC) metric to dynamically assess the reliability of each modality at the instance level under the current input, leveraging conditional probability circuits. Fusion is then adaptively performed via KL divergence, enabling both precision and interpretability. Experimental results demonstrate that C²MF achieves up to a 29% accuracy improvement over static baselines in high-noise and modality-conflict scenarios, while preserving the inherent interpretability of probabilistic circuits.

This work addresses the challenge of multimodal sentiment analysis in real-world scenarios, where performance is often hindered by missing observational data and the lack of explicit modeling of modality reliability in existing methods, leading to reliability mismatch and propagation bias. To overcome these limitations, the authors propose the Modality Reliability-aware Collaborative Fusion (MRCF) framework, which introduces, for the first time, a sample-level modality reliability assessment mechanism that integrates intra-modality quality cues with cross-modality semantic consistency. MRCF dynamically regulates multimodal information flow through a reliability-aware branch, a reliability-guided interaction mechanism, and a calibration-based fusion module. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that the proposed approach significantly enhances model robustness and accuracy under incomplete observational conditions.

This work addresses the unreliability of individual modalities in multimodal intent recognition, which often arises from noise, missing data, semantic conflicts, or excessive dominance, and notes that existing methods lack mechanisms to assess modality reparability. To overcome this limitation, the paper proposes PRIME, a novel framework that introduces, for the first time, a label-free reparability diagnosis mechanism operating without explicit modality reliability annotations. PRIME employs a closed-loop pipeline to jointly diagnose, repair, and re-evaluate modality reliability at the sample level. It integrates heteroscedastic uncertainty modeling with multidimensional diagnostic signals—such as cognitive disagreement and cross-modal consistency—to drive a prototype-conditioned variational restoration module that reconstructs missing or corrupted evidence using complementary modalities. Robustness is further enhanced through inverse-variance fusion. Experiments demonstrate that PRIME achieves competitive performance on standard benchmarks and significantly outperforms state-of-the-art methods under various perturbations, including modality missingness, noise, conflict, and imbalance.

This work addresses the opacity of reasoning processes in multimodal large language models (MLLMs) by identifying a novel diagnostic failure mode—“modality disruption”: high-confidence unimodal errors dominate multimodal fusion decisions, thereby suppressing corroborative evidence from other modalities. To diagnose this phenomenon, we propose a lightweight, model-agnostic framework that treats each modality as an independent agent. The framework integrates candidate label generation, self-evaluation prompting, and aggregation-based fusion to enable interpretable auditing of both modality-specific contributions and disruptive behaviors. Crucially, it is the first method to systematically characterize the dynamics of multimodal conflict. Evaluated on sentiment analysis benchmarks, it effectively disentangles data bias from intrinsic model deficiencies. Our approach provides a transferable, principled diagnostic tool for assessing the reasoning reliability of MLLMs.

This work addresses the tendency of multimodal large language models to exhibit overconfidence and visual hallucinations when faced with insufficient visual evidence or semantic conflicts, compounded by the absence of instance-level reliability assessment mechanisms. To enhance model calibration without requiring retraining, the authors propose a retrieval-augmented, reliability-aware reasoning framework. This approach leverages an external visual evidence bank, integrating pretrained visual embeddings, normalized nearest-neighbor retrieval, and multidimensional reliability metrics—including similarity strength, category consistency, evidence boundary, and entropy-based uncertainty—alongside a decision gating mechanism that dynamically chooses to accept, respond cautiously, or abstain from answering. Evaluated on ImageNet-100, the method improves the accuracy of accepted predictions from 85.84% to 88.88% at a coverage rate of 89.04%, while reducing hallucination error rates by 3.04 percentage points.

This work addresses the challenge in multimodal sentiment analysis that different modalities exhibit significantly varying reliability at the utterance level due to factors such as occlusion, noise, or transcription errors, which can adversely affect traditional fusion approaches. To mitigate this issue, the authors propose MRUF, a method that enables adaptive fusion through multi-granularity routing and uncertainty-aware calibration. Specifically, MRUF incorporates modality importance supervision derived from leave-one-out error increases, an inverse-variance reweighting mechanism for uncertainty-based gating, and a modality-invariant contrastive alignment strategy. Experimental results demonstrate that MRUF consistently outperforms strong baselines on both aligned and unaligned settings of the CMU-MOSI and CMU-MOSEI datasets, effectively down-weighting contributions from high-uncertainty modalities and thereby enhancing model robustness.

This work addresses the problem of miscalibrated confidence in multimodal fusion caused by missing modalities. It proposes Modal-Conditional Conformal Fusion (MCCF), a method that integrates evidential deep learning with Dempster–Shafer theory. During training, MCCF simulates missing modalities via random modality dropout, allowing absent modalities to contribute vacuous evidence automatically. By incorporating Mondrian conformal prediction, MCCF provides finite-sample coverage guarantees for any non-empty subset of available modalities at test time—without requiring imputation. To the best of our knowledge, MCCF is the first approach to achieve formally calibrated uncertainty under arbitrary modality availability, while also decomposing evidence to yield modality-level nullity scores for uncertainty attribution. Experiments on synthetic data and three real-world benchmarks demonstrate that MCCF consistently attains target coverage, substantially narrows the coverage gap between full and partial modalities, and preserves predictive accuracy.

This work addresses the challenge in multimodal fusion where unknown modality reliability can lead to over-weighting of unreliable or redundant modalities, thereby degrading predictive performance. To mitigate this, the authors propose TIER-MoE, a novel mixture-of-experts architecture that, for the first time, jointly incorporates sample-level conditional modality risk—estimated via leave-one-out prediction loss—and expert subspace compatibility to enable risk-aware sparse modality-to-expert routing. The model retains a shared multimodal pathway to preserve complementary information across modalities. Evaluated on four biomedical multimodal datasets, TIER-MoE consistently outperforms state-of-the-art methods, achieving significant improvements in both Macro-F1 and Brier score, while also demonstrating strong zero-shot external generalization capabilities.

Existing multimodal fusion models often lack robustness under noisy or uninformative data and fail to dynamically assess data quality or produce reliable confidence estimates, limiting their applicability in high-stakes clinical settings. To address these challenges, this work proposes the Adaptive Confidence-weighted Extension (ACE) framework, which uniquely integrates intra-modality correlation–driven complementary modality generation with a dual-level dynamic confidence mechanism. This enables adaptive weighting of modality reliability and outputs a global trust score. Evaluated on four multi-omics datasets—BRCA, KIPAN, LGG, and ROSMAP—ACE significantly outperforms current methods, achieving notable improvements in both classification accuracy and confidence calibration, thereby enhancing model robustness and clinical trustworthiness.