Recalibrating the Compass: Integrating Large Language Models into Classical Research Methods

2025-05-26

AI Summary

This paper addresses the challenge of integrating large language models (LLMs) into communication and social science research without displacing foundational quantitative methods—namely content analysis, survey research, and experimental design. To mitigate LLM limitations concerning validity, bias, and interpretability, the study systematically reconceptualizes Lasswell’s “five Ws” framework, introducing three novel paradigms: LLM-driven explanatory variation modeling, dynamic audience trajectory simulation, and counterfactual experimental design. Methodologically, it synthesizes computational social science, explainable AI (XAI), generative text modeling, and simulation-based reasoning, while embedding traditional reliability/validity assessment and critical AI auditing. The contribution is a rigor-preserving methodological pathway that extends empirical boundaries, offering social scientists an innovative, reproducible, and ethically reflexive operational framework for LLM-augmented research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

This paper examines how large language models (LLMs) are transforming core quantitative methods in communication research in particular, and in the social sciences more broadly-namely, content analysis, survey research, and experimental studies. Rather than replacing classical approaches, LLMs introduce new possibilities for coding and interpreting text, simulating dynamic respondents, and generating personalized and interactive stimuli. Drawing on recent interdisciplinary work, the paper highlights both the potential and limitations of LLMs as research tools, including issues of validity, bias, and interpretability. To situate these developments theoretically, the paper revisits Lasswell's foundational framework --"Who says what, in which channel, to whom, with what effect?"-- and demonstrates how LLMs reconfigure message studies, audience analysis, and effects research by enabling interpretive variation, audience trajectory modeling, and counterfactual experimentation. Revisiting the metaphor of the methodological compass, the paper argues that classical research logics remain essential as the field integrates LLMs and generative AI. By treating LLMs not only as technical instruments but also as epistemic and cultural tools, the paper calls for thoughtful, rigorous, and imaginative use of LLMs in future communication and social science research.

Problem

• Integrating LLMs into classical social science research methods
• Exploring LLMs' potential and limitations in communication research
• Reconfiguring message studies and audience analysis using LLMs

Innovation

• LLMs enhance text coding and interpretation
• LLMs simulate dynamic survey respondents
• LLMs generate personalized interactive stimuli