Uncovering Latent Arguments in Social Media Messaging by Employing LLMs-in-the-Loop Strategy

2024-04-16

arXiv.org

AI Summary

Identifying implicit arguments in social media discourse on contentious topics—such as climate change and COVID-19 vaccines—remains challenging due to their dynamic nature, semantic subtlety, and lack of fine-grained, temporally aware modeling. Method: This paper proposes an LLM-in-the-Loop framework that integrates prompt-driven argument generation using GPT-4, topic-constrained clustering, human-in-the-loop validation, and cross-event temporal comparative analysis. Contribution/Results: It is the first work to deeply embed large language models into the end-to-end argument mining pipeline, overcoming the traditional trade-off between adaptability and semantic granularity inherent in supervised or unsupervised approaches. Evaluated on a real-world Facebook ad dataset, the method extracts hundreds of high signal-to-noise implicit arguments. It systematically uncovers audience-targeting disparities and event-driven discursive evolution patterns—enabling interpretable, iterative modeling of argument dynamics across time and contexts.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Knowledge Representation and Reasoning: Argumentation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Topic discovery and tracking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The widespread use of social media has led to a surge in popularity for automated methods of analyzing public opinion. Supervised methods are adept at text categorization, yet the dynamic nature of social media discussions poses a continual challenge for these techniques due to the constant shifting of the focus. On the other hand, traditional unsupervised methods for extracting themes from public discourse, such as topic modeling, often reveal overarching patterns that might not capture specific nuances. Consequently, a significant portion of research into social media discourse still depends on labor-intensive manual coding techniques and a human-in-the-loop approach, which are both time-consuming and costly. In this work, we study the problem of discovering arguments associated with a specific theme. We propose a generic LLMs-in-the-Loop strategy that leverages the advanced capabilities of Large Language Models (LLMs) to extract latent arguments from social media messaging. To demonstrate our approach, we apply our framework to contentious topics. We use two publicly available datasets: (1) the climate campaigns dataset of 14k Facebook ads with 25 themes and (2) the COVID-19 vaccine campaigns dataset of 9k Facebook ads with 14 themes. Furthermore, we analyze demographic targeting and the adaptation of messaging based on real-world events.

Problem

• Social Media Analysis
• Controversial Topics
• Hidden Sentiment Identification

Innovation

• LLMs-in-the-Loop
• Social Media Analysis
• Controversial Topics