Identifying Subgroup and Context Effects in Conjoint Experiments

2025-10-02

AI Summary

Reliable inference of main and interaction effects in joint experiments faces challenges including low statistical power and difficulty identifying effect heterogeneity. This paper proposes a novel black-box inference framework that integrates sparse regression, Bayesian modeling, post-hoc calibration, and machine learning algorithms to improve estimation accuracy while preserving computational efficiency. The framework systematically evaluates the performance of multiple state-of-the-art methods under diverse data-generating mechanisms and validates its robustness through large-scale simulations. Results demonstrate that the proposed method substantially enhances detection power for weak interaction effects, particularly in subgroup and contextual variable analyses. By bridging theoretical rigor with practical feasibility, this work provides social scientists with a new tool for causal heterogeneity inference—offering improved reliability, flexibility, and scalability compared to existing approaches.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Conjoint experiments have become central to survey research in political science and related fields because they allow researchers to study preferences across multiple attributes simultaneously. Beyond estimating main effects, scholars increasingly analyze heterogeneity through subgroup analysis and contextual variables, raising methodological challenges in detecting and interpreting interaction effects. Statistical power constraints, common in survey experiments, further complicate this task. This paper addresses the question: how can both main and interaction effects be reliably inferred in conjoint studies? We contribute in two ways. First, we conduct a systematic evaluation of leading approaches, including post-hoc corrections, sparse regression methods, and Bayesian models, across simulation regimes that vary sparsity, noise, and data availability. Second, we propose a novel black-box inference framework that leverages machine learning to recover main and interaction effects in conjoint experiments. Our approach balances computational efficiency with accuracy, providing a practical tool for researchers studying heterogeneous effects.

Problem

• Detecting interaction effects in conjoint experiments
• Addressing statistical power constraints in survey research
• Recovering main and interaction effects using machine learning

Innovation

• Proposes black-box inference framework using machine learning
• Evaluates post-hoc corrections and sparse regression methods
• Balances computational efficiency with accuracy for effects