Are LLM-Powered Social Media Bots Realistic?

2025-08-01

AI Summary

This study investigates the realism of LLM-driven social media bots at both network-structural and linguistic-behavioral levels. We propose a multidimensional simulation framework integrating human-crafted persona design, network science modeling, and LLM-generated content to systematically synthesize bot identities, tweets, and interaction patterns. Realism is quantitatively assessed via topological metrics (e.g., degree distribution, clustering coefficient) and linguistic features (e.g., lexical frequency, syntactic complexity, sentiment consistency). Experimental results reveal that LLM bot networks differ significantly from both real human networks and rule-based bots: they exhibit excessive sparsity and absent community structure in their social graphs, while their language displays high coherence but low lexical diversity and anomalous sentiment dynamics. These findings expose a structural “distortion” inherent to LLM bots—characterized by decoupled network topology and linguistic behavior—thereby establishing an interpretable, dual-dimensional (topological + linguistic) foundation for next-generation bot detection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As Large Language Models (LLMs) become more sophisticated, there is a possibility to harness LLMs to power social media bots. This work investigates the realism of generating LLM-Powered social media bot networks. Through a combination of manual effort, network science and LLMs, we create synthetic bot agent personas, their tweets and their interactions, thereby simulating social media networks. We compare the generated networks against empirical bot/human data, observing that both network and linguistic properties of LLM-Powered Bots differ from Wild Bots/Humans. This has implications towards the detection and effectiveness of LLM-Powered Bots.

Problem

• Investigating realism of LLM-powered social media bots
• Comparing bot networks with empirical human/bot data
• Assessing detection and effectiveness of LLM-powered bots

Innovation

• Using LLMs to create synthetic bot personas
• Combining network science with LLM-generated content
• Comparing LLM bot networks with empirical data