Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest

2026-04-20

AI Summary

This study addresses the lack of systematic evaluation of large language models (LLMs) in multitask social media analysis, particularly concerning authorship verification, content generation, and user attribute inference. The authors construct a unified evaluation framework to assess prominent models—including GPT-4, GPT-4o, GPT-3.5-Turbo, Gemini 1.5 Pro, DeepSeek-V3, Llama 3.2, and BERT—on a Twitter dataset. They introduce a novel sampling strategy to mitigate “seen-data” bias and establish a reproducible benchmark by integrating attribute annotations based on IAB Tech Lab and U.S. Standard Occupational Classification (SOC) standards with real-user studies. Their findings reveal significant performance disparities across models in generating authentic content, generalizing verification tasks, and inferring user attributes, offering new insights for LLM-driven social media analysis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In this study, we present the first comprehensive evaluation of modern LLMs - including GPT-4, GPT-4o, GPT-3.5-Turbo, Gemini 1.5 Pro, DeepSeek-V3, Llama 3.2, and BERT - across three core social media analytics tasks on a Twitter (X) dataset: (I) Social Media Authorship Verification, (II) Social Media Post Generation, and (III) User Attribute Inference. For the authorship verification, we introduce a systematic sampling framework over diverse user and post selection strategies and evaluate generalization on newly collected tweets from January 2024 onward to mitigate "seen-data" bias. For post generation, we assess the ability of LLMs to produce authentic, user-like content using comprehensive evaluation metrics. Bridging Tasks I and II, we conduct a user study to measure real users' perceptions of LLM-generated posts conditioned on their own writing. For attribute inference, we annotate occupations and interests using two standardized taxonomies (IAB Tech Lab 2023 and 2018 U.S. SOC) and benchmark LLMs against existing baselines. Overall, our unified evaluation provides new insights and establishes reproducible benchmarks for LLM-driven social media analytics. The code and data are provided in the supplementary material and will also be made publicly available upon publication.

Problem

• Large Language Models
• Social Media Analytics
• Authorship Verification
• Post Generation
• User Attribute Inference

Innovation

• Large Language Models
• Social Media Analytics
• Authorship Verification
• User Attribute Inference
• Multi-task Evaluation