Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?

2025-02-11

AI Summary

This study investigates whether large language models (LLMs) are susceptible to “spin”—author-driven rhetorical distortion of research findings—in medical literature, and assesses their potential to propagate interpretive bias. We constructed a manually annotated dataset of spin-labeled medical abstracts and conducted a systematic evaluation across 22 state-of-the-art LLMs using multi-model benchmarking, controlled prompting experiments, and bias analysis of generated summaries. Results reveal, for the first time, that LLMs are on average more vulnerable to spin than human readers; however, certain models demonstrate emergent spin-detection capability. Critically, targeted prompt engineering—including explicit instructions and metacognitive prompts—significantly mitigates spin-induced distortion in generated summaries. These findings expose a key vulnerability—and concurrent plasticity—of LLMs in evidence-based medicine applications. The work provides both methodological foundations and empirical evidence to enhance the reliability of LLMs in clinical decision support.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Medical research faces well-documented challenges in translating novel treatments into clinical practice. Publishing incentives encourage researchers to present"positive"findings, even when empirical results are equivocal. Consequently, it is well-documented that authors often spin study results, especially in article abstracts. Such spin can influence clinician interpretation of evidence and may affect patient care decisions. In this study, we ask whether the interpretation of trial results offered by Large Language Models (LLMs) is similarly affected by spin. This is important since LLMs are increasingly being used to trawl through and synthesize published medical evidence. We evaluated 22 LLMs and found that they are across the board more susceptible to spin than humans. They might also propagate spin into their outputs: We find evidence, e.g., that LLMs implicitly incorporate spin into plain language summaries that they generate. We also find, however, that LLMs are generally capable of recognizing spin, and can be prompted in a way to mitigate spin's impact on LLM outputs.

Problem

• LLMs susceptibility to spin
• Impact of spin on medical decisions
• Mitigating spin in LLM outputs

Innovation

• LLMs evaluate spin susceptibility
• LLMs generate spin-influenced summaries
• Prompting mitigates spin impact