BTS: A Comprehensive Benchmark for Tie Strength Prediction

2024-10-24

AI Summary

Tie strength (TS) prediction in online social networks has long suffered from the absence of ground-truth labels, inconsistent evaluation protocols, and poor model generalizability. To address these challenges, we introduce BTS—the first comprehensive benchmark for TS prediction—featuring seven standardized pseudo-label generation techniques and a novel framework for evaluating pseudo-label quality based on relational resilience. We systematically analyze data distribution and label correlation across multiple networks, integrating social network analysis, statistical hypothesis testing, and cross-model evaluation. Our unified empirical study evaluates state-of-the-art methods on three real-world networks, revealing critical impacts of experimental setup and metric selection on reported performance. All datasets, code, and evaluation scripts are publicly released to establish a reproducible, comparable, and standardized foundation for future TS research.

Technology Category

• Machine Learning: Evaluation and Analysis
• Application Domains: Humanities & Computational Social Science
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

The rapid rise of online social networks underscores the need to understand the heterogeneous strengths of online relationships. Yet, efforts to assess tie strength (TS) are hindered by the lack of ground-truth labels, differing research perspectives, and limited model performance in real-world settings. To address this gap, we introduce BTS, a comprehensive Benchmark for Tie Strength prediction, aiming to establish a standardized foundation for evaluating and advancing TS prediction methodologies. Specifically, our contributions are: TS Pseudo-Label Techniques -- we categorize TS into seven standardized pseudo-labeling techniques based on prior literature; TS Dataset Collection -- we present a representative collection of three social networks and perform data analysis by investigating the class distributions and correlations across the generated pseudo-labels; TS Pseudo-Label Evaluation Framework -- we propose a standardized framework to evaluate the pseudo-label quality from the perspective of tie resilience; Benchmarking -- we evaluate existing tie strength prediction model performance using the BTS dataset collection, exploring the effects of different experiment settings, models, and evaluation criteria on the results. Furthermore, we derive key insights to enhance existing methods and shed light on promising directions for future research in this domain. The BTS dataset collection, along with the curation codes, and experimental scripts are all available at: https://github.com/XueqiC/Awesome-Tie-Strength-Prediction.

Problem

• Lack of ground-truth labels for tie strength prediction
• Differing research perspectives on tie strength assessment
• Limited model performance in real-world social networks

Innovation

• Standardized pseudo-labeling techniques for TS
• Dataset collection from three social networks
• Evaluation framework for pseudo-label quality