A Comparative Analysis of Transformer Models in Social Bot Detection

2025-09-18

AI Summary

Social media platforms face growing threats from LLM-generated fake accounts, which severely undermine discourse authenticity—necessitating robust social bot detection methods. This paper systematically compares encoder-only (e.g., BERT) and decoder-only (e.g., LLaMA, GPT) Transformer architectures for bot detection, establishing a unified evaluation framework across four dimensions: classification accuracy, adversarial robustness, task adaptability, and interpretability. Experimental results show that encoder models significantly outperform decoder models in accuracy and stability under adversarial perturbations; conversely, decoder models—despite slightly lower accuracy—exhibit superior generalization in few-shot transfer learning, prompt alignment, and behavioral interpretability, leveraging their generative priors and instruction-tuning capabilities. To our knowledge, this is the first study to empirically reveal the complementary strengths of these two architectural paradigms, thereby providing principled guidance for architecture selection and inspiring hybrid modeling strategies for social bot detection in the generative AI era.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Social media has become a key medium of communication in today's society. This realisation has led to many parties employing artificial users (or bots) to mislead others into believing untruths or acting in a beneficial manner to such parties. Sophisticated text generation tools, such as large language models, have further exacerbated this issue. This paper aims to compare the effectiveness of bot detection models based on encoder and decoder transformers. Pipelines are developed to evaluate the performance of these classifiers, revealing that encoder-based classifiers demonstrate greater accuracy and robustness. However, decoder-based models showed greater adaptability through task-specific alignment, suggesting more potential for generalisation across different use cases in addition to superior observa. These findings contribute to the ongoing effort to prevent digital environments being manipulated while protecting the integrity of online discussion.

Problem

• Comparing encoder and decoder transformers for social bot detection
• Evaluating bot detection models' accuracy, robustness, and adaptability
• Preventing digital manipulation while protecting online discussion integrity

Innovation

• Encoder-based classifiers for accuracy
• Decoder-based models for adaptability
• Comparative analysis of transformer models