Improving the Accuracy of Amortized Model Comparison with Self-Consistency

2025-12-16

AI Summary

Amortized Bayesian model comparison is vulnerable to extrapolation bias under model misspecification—particularly when observed data lie outside the training distribution—rendering neural surrogate models unreliable. To address this, we propose a robust self-consistency (SC)-based training framework and provide the first systematic evaluation of SC’s differential benefits across diverse amortized model comparison paradigms. Our contributions are threefold: (1) We empirically demonstrate that parameter-posterior–driven approaches substantially outperform direct marginal likelihood approximations; (2) we derive practical deployment guidelines for misspecified settings; and (3) across two synthetic and two real-world benchmarks, SC-augmented parameter-posterior methods significantly improve model selection accuracy and maintain robustness even under severe misspecification, whereas likelihood-free alternatives exhibit only marginal and unstable gains.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Amortized Bayesian inference (ABI) offers fast, scalable approximations to posterior densities by training neural surrogates on data simulated from the statistical model. However, ABI methods are highly sensitive to model misspecification: when observed data fall outside the training distribution (generative scope of the statistical models), neural surrogates can behave unpredictably. This makes it a challenge in a model comparison setting, where multiple statistical models are considered, of which at least some are misspecified. Recent work on self-consistency (SC) provides a promising remedy to this issue, accessible even for empirical data (without ground-truth labels). In this work, we investigate how SC can improve amortized model comparison conceptualized in four different ways. Across two synthetic and two real-world case studies, we find that approaches for model comparison that estimate marginal likelihoods through approximate parameter posteriors consistently outperform methods that directly approximate model evidence or posterior model probabilities. SC training improves robustness when the likelihood is available, even under severe model misspecification. The benefits of SC for methods without access of analytic likelihoods are more limited and inconsistent. Our results suggest practical guidance for reliable amortized Bayesian model comparison: prefer parameter posterior-based methods and augment them with SC training on empirical datasets to mitigate extrapolation bias under model misspecification.

Problem

• Improves robustness of amortized model comparison under misspecification
• Compares parameter posterior-based methods to direct evidence approximation
• Investigates self-consistency training for mitigating extrapolation bias in empirical data

Innovation

• Self-consistency training enhances robustness under model misspecification
• Parameter posterior-based methods outperform direct evidence approximation
• SC mitigates extrapolation bias in amortized Bayesian model comparison