SocialX: A Modular Platform for Multi-Source Big Data Research in Indonesia

2026-03-27

AI Summary

This study addresses the persistent challenge of data fragmentation in Indonesian multi-source big data research, where heterogeneous data formats, access protocols, and noise characteristics across platforms often necessitate redundant development of collection and analysis pipelines. To overcome this, the authors propose a modular, source-agnostic unified processing platform featuring a three-layer decoupled architecture—comprising data ingestion, language-aware preprocessing, and pluggable analytics—alongside a lightweight task orchestration mechanism. This design enables seamless integration of new data sources, preprocessing methods, or analytical tools. The platform, publicly available at https://www.socialx.id, has been validated through representative workflows, demonstrating a significant reduction in the barrier to entry for multi-source big data research while enhancing scalability and reusability.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Application Domains: Humanities & Computational Social Science
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Web Mining and Content Analysis: Web data integration and cleaning
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance

Abstract

Big data research in Indonesia is constrained by a fundamental fragmentation: relevant data is scattered across social media, news portals, e-commerce platforms, review sites, and academic databases, each with different formats, access methods, and noise characteristics. Researchers must independently build collection pipelines, clean heterogeneous data, and assemble separate analysis tools, a process that often overshadows the research itself. We present SocialX, a modular platform for multi-source big data research that integrates heterogeneous data collection, language-aware preprocessing, and pluggable analysis into a unified, source-agnostic pipeline. The platform separates concerns into three independent layers (collection, preprocessing, and analysis) connected by a lightweight job-coordination mechanism. This modularity allows each layer to grow independently: new data sources, preprocessing methods, or analysis tools can be added without modifying the existing pipeline. We describe the design principles that enable this extensibility, detail the preprocessing methodology that addresses challenges specific to Indonesian text across registers, and demonstrate the platform's utility through a walkthrough of a typical research workflow. SocialX is publicly accessible as a web-based platform at https://www.socialx.id.

Problem

• big data
• data fragmentation
• heterogeneous data
• Indonesia
• multi-source data

Innovation

• modular architecture
• multi-source big data
• language-aware preprocessing
• extensible pipeline
• Indonesian NLP