TwiUSD: A Benchmark Dataset and Structure-Aware LLM Framework for User Stance Detection

2025-06-16

AI Summary

User Stance Detection (UserSD) has long suffered from the absence of high-quality benchmark datasets that jointly capture linguistic and social structural signals. To address this, we introduce TwiUSD—the first large-scale, manually annotated UserSD benchmark—comprising 16,211 users and 47,757 tweets with explicit follower-followee relationships, enabling the first explicit modeling of social attention graphs. To tackle the heterogeneity between textual and graph-structured data, we propose MRFG, a structure-aware framework integrating LLM-driven relevance filtering with topology-aware multi-scale feature routing, which adaptively coordinates GNNs and MLPs for joint representation learning. Extensive experiments demonstrate that MRFG significantly outperforms state-of-the-art PLMs, graph-based models, and LLM prompting methods under both in-target and cross-target evaluation settings. Our results empirically validate that explicit modeling of social graph structure is critical for robust UserSD.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Social recommender systems and personalization for the social Web
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

User-level stance detection (UserSD) remains challenging due to the lack of high-quality benchmarks that jointly capture linguistic and social structure. In this paper, we introduce TwiUSD, the first large-scale, manually annotated UserSD benchmark with explicit followee relationships, containing 16,211 users and 47,757 tweets. TwiUSD enables rigorous evaluation of stance models by integrating tweet content and social links, with superior scale and annotation quality. Building on this resource, we propose MRFG: a structure-aware framework that uses LLM-based relevance filtering and feature routing to address noise and context heterogeneity. MRFG employs multi-scale filtering and adaptively routes features through graph neural networks or multi-layer perceptrons based on topological informativeness. Experiments show MRFG consistently outperforms strong baselines (including PLMs, graph-based models, and LLM prompting) in both in-target and cross-target evaluation.

Problem

• Lack of high-quality benchmarks for user stance detection
• Noise and context heterogeneity in stance detection models
• Need for structure-aware frameworks integrating social links

Innovation

• Large-scale annotated dataset with social links
• LLM-based relevance filtering for noise reduction
• Adaptive feature routing via graph networks