Unraveling SITT: Social Influence Technique Taxonomy and Detection with LLMs

2025-05-29

AI Summary

This work addresses the challenge that large language models (LLMs) struggle to identify implicit social influence tactics in text. We propose SITT—the first empirically grounded, fine-grained taxonomy of social influence techniques—comprising 58 tactics organized into nine categories. Leveraging 746 bilingual dialogues annotated collaboratively by domain experts, we construct the first high-quality, cross-lingual, multi-expert-validated SITT benchmark. Using a hierarchical multi-label classification paradigm, we systematically evaluate leading LLMs—including GPT-4o, Claude 3.5, and Llama-3.1—revealing low category-level F1 scores (max 0.45), especially for context-sensitive tactics; domain-specific fine-tuning significantly improves robustness. Our contributions are threefold: (1) the first fine-grained, empirically validated taxonomy of social influence techniques; (2) the first cross-lingual, expert-verified SITT benchmark; and (3) empirical evidence characterizing fundamental limitations of current LLMs in recognizing subtle, pragmatic social influence strategies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In this work we present the Social Influence Technique Taxonomy (SITT), a comprehensive framework of 58 empirically grounded techniques organized into nine categories, designed to detect subtle forms of social influence in textual content. We also investigate the LLMs ability to identify various forms of social influence. Building on interdisciplinary foundations, we construct the SITT dataset -- a 746-dialogue corpus annotated by 11 experts in Polish and translated into English -- to evaluate the ability of LLMs to identify these techniques. Using a hierarchical multi-label classification setup, we benchmark five LLMs, including GPT-4o, Claude 3.5, Llama-3.1, Mixtral, and PLLuM. Our results show that while some models, notably Claude 3.5, achieved moderate success (F1 score = 0.45 for categories), overall performance of models remains limited, particularly for context-sensitive techniques. The findings demonstrate key limitations in current LLMs' sensitivity to nuanced linguistic cues and underscore the importance of domain-specific fine-tuning. This work contributes a novel resource and evaluation example for understanding how LLMs detect, classify, and potentially replicate strategies of social influence in natural dialogues.

Problem

• Detecting subtle social influence techniques in text
• Evaluating LLMs' ability to identify influence strategies
• Assessing model performance on context-sensitive linguistic cues

Innovation

• Hierarchical multi-label classification for influence detection
• Domain-specific fine-tuning of LLMs for sensitivity
• Expert-annotated multilingual dataset for evaluation