Statistical significance in choice modelling: computation, usage and reporting

2025-06-06

AI Summary

This paper identifies widespread misuse of statistical significance in choice modeling: overreliance on the 95% confidence threshold, conflation of *p*-values with effect size, superficial reporting of asterisk-based uncertainty indicators, and neglect of behavioral or policy relevance. To address these issues, the authors propose—first in the literature—a dual-dimension evaluation framework integrating “statistical significance” and “behavioral/policy significance.” The framework systematically unifies classical statistical inference (hypothesis testing, confidence intervals), sensitivity analysis, and uncertainty quantification, clarifying conceptual distinctions and standardizing the computation and reporting of uncertainty measures. By shifting focus from statistical significance alone to decision-relevant, transparent, and empirically rigorous evaluation, the framework advances methodological practice in choice modeling and provides actionable guidelines for improving empirical research standards. (149 words)

Technology Category

• Reasoning under Uncertainty: Decision/Utility Theory
• Machine Learning: Calibration & Uncertainty Quantification
• Game Theory and Economic Paradigms: Social Choice / Voting

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We argue that, as in many other areas of science, there is an over-reliance on 95% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using p-values and even more so with star measures. The paper provides a precise discussion on the computation of measures of uncertainty and confidence intervals, discusses the use of statistical tests, and also stresses the importance of considering behavioural or policy significance in addition to statistical significance.

Problem

• Over-reliance on 95% confidence levels in choice modelling
• Misunderstandings of statistical significance meaning and usage
• Lack of precision in reporting uncertainty measures like p-values

Innovation

• Precise computation of uncertainty measures
• Discussion on statistical tests usage
• Emphasis on behavioral significance consideration