LLM-Assisted Stance Detection in Scientific Discourse: A Test Case in Bayesian Cognitive Science

2026-06-13

AI Summary

This study addresses the challenge of automatically identifying authors’ stances—realist versus instrumentalist—toward Bayesian models in scientific texts, a task that is theoretically laden, expressed implicitly, and difficult to annotate manually. To tackle this, the authors propose an expert-guided, diagnosis-driven prompt optimization framework. Leveraging a theory-informed coding manual, expert annotations, and iterative diagnostic prompt searches, they develop a unified zero-shot prompt for GPT-5.1, Claude Sonnet 4.6, and Gemini 3 Pro Preview. The approach achieves strong intercoder reliability (0.76 overall, 0.78 at the citation level) and high article-level stance ranking stability (r = 0.96–0.97). Furthermore, it reveals that research in low-level perceptual and motor domains exhibits a significantly stronger realist tendency (d = 0.60, p < .001), demonstrating the feasibility and reliability of large language models in highly interpretive qualitative analysis.

Abstract

Qualitative coding is central to social science, but expert annotation is difficult to scale. LLMs offer a possible extension, yet require careful validation when the target construct is interpretive, theoretically loaded, and only indirectly expressed. We study this problem in a difficult case: detecting whether authors treat Bayesian models as descriptions of mental and neural mechanisms (realism) or as useful mathematical tools (instrumentalism). Our method combines a theory-driven codebook, expert-coded reference annotations, a diagnostic-gated prompt-optimization search yielding a shared zero-shot prompt for three frontier LLMs (GPT-5.1, Claude Sonnet 4.6, Gemini 3 Pro Preview), and multi-rater reliability analysis. The final prompt achieved a held-out combined reliability score of 0.76 (harmonic mean of ICC = 0.79 and $α$ = 0.74), with all diagnostics satisfied. Deployed on 6,858 quotes from 210 articles, the three LLMs reached substantial quote-level agreement (ICC = 0.80; $α$ = 0.76; combined = 0.78) and near-perfect article-level rank stability ($r$ = 0.96-0.97 across rater pairs). The corpus was predominantly weakly realist, but article-level stances were rarely uniform: only 1.4% of articles used a single band, while 59.5% spanned four or more. Low-level perception/motor articles scored 8.8 Realism points higher than high-level cognition articles ($p < .001$, $d = 0.60$), quantifying a long-held qualitative intuition. We present this as an expert-led case study; the framework is intended to generalize to similar theoretically demanding tasks, not to all qualitative analysis.

Problem

• stance detection
• Bayesian cognitive science
• realism vs instrumentalism
• scientific discourse
• qualitative coding

Innovation

• LLM-assisted stance detection
• zero-shot prompting
• diagnostic-gated prompt optimization
• multi-rater reliability
• Bayesian cognitive science