Towards Polyglot Data Processing in Social Networks using the Hadoop-Spark Ecosystem

2025-01-20

Artificial Intelligence and Big Data Trends 2025

AI Summary

To address the challenge of processing multi-source, heterogeneous data in social networks, this paper proposes a unified batch-stream-graph analytics framework built upon the Hadoop-Spark ecosystem. The method systematically integrates Hive (for SQL-based batch processing), HBase (for low-latency key-value lookups), and GraphX (for scalable graph computation) under a single Spark execution layer. It supports three core analytical tasks: user influence assessment, high-frequency term statistics, and community relationship mining. Leveraging HDFS for distributed storage, YARN for resource orchestration, and multi-language APIs, the framework achieves loosely coupled integration of computation and storage. End-to-end experiments on real-world social datasets demonstrate that the hybrid architecture accelerates complex relational analysis by 1.8–3.2× compared to single-component baselines, significantly improving both processing efficiency and system flexibility.

Technology Category

• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Machine Learning: Graph-based Machine Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Social Networks and Social Media: Social media analysis through the lenses of networks
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

This article explores the use of the Hadoop-Spark ecosystem for social media data processing, adopting a polyglot approach with the integration of various computation and storage technologies, such as Hive, HBase and GraphX. We discuss specific tasks involved in processing social network data, such as calculating user influence, counting the most frequent terms in messages and identifying social relationships among users and groups. We conducted a series of empirical performance assessments, focusing on executing selected tasks and measuring their execution time within the Hadoop-Spark cluster. These insights offer a detailed quantitative analysis of the performance efficiency of the ecosystem tools. We conclude by highlighting the potential of the Hadoop-Spark ecosystem tools for advancing research in social networks and related fields.

Problem

• Processing social media data with Hadoop-Spark ecosystem
• Measuring performance of polyglot data tasks in clusters
• Analyzing user influence and relationships in networks

Innovation

• Hadoop-Spark ecosystem for polyglot data processing
• Integration of Hive, HBase, and GraphX technologies
• Empirical performance assessments in social networks