hadoop ecosystem

To address the challenge of processing multi-source, heterogeneous data in social networks, this paper proposes a unified batch-stream-graph analytics framework built upon the Hadoop-Spark ecosystem. The method systematically integrates Hive (for SQL-based batch processing), HBase (for low-latency key-value lookups), and GraphX (for scalable graph computation) under a single Spark execution layer. It supports three core analytical tasks: user influence assessment, high-frequency term statistics, and community relationship mining. Leveraging HDFS for distributed storage, YARN for resource orchestration, and multi-language APIs, the framework achieves loosely coupled integration of computation and storage. End-to-end experiments on real-world social datasets demonstrate that the hybrid architecture accelerates complex relational analysis by 1.8–3.2× compared to single-component baselines, significantly improving both processing efficiency and system flexibility.

To address the challenges of heterogeneous engine coordination, semantic fragmentation, and inefficient scheduling in multilingual big data processing, this paper proposes the first unified polyglot data processing framework built on the Hadoop ecosystem. The framework leverages YARN as its resource management foundation and integrates HDFS, Spark, Flink, Kafka, HBase, and a custom DSL-driven hybrid execution engine. It introduces a novel semantics-aware component coordination mechanism and scenario-adaptive orchestration strategy. Evaluated on real-world smart city and social network workloads, the framework achieves an average 37% reduction in end-to-end latency and a 2.1× improvement in resource utilization—demonstrating the efficacy of cross-engine semantic alignment and dynamic collaborative scheduling. This work establishes a systematic architectural paradigm and provides empirical validation for evolving the Hadoop ecosystem into a unified, multi-paradigm, polyglot data processing platform.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

Machine learning teams face significant challenges in multi-workflow concurrent environments, including redundant intermediate data storage, inefficient cross-pipeline sharing, and high collaboration overhead. To address these issues, this paper proposes and implements a data virtualization service architecture tailored for ML workflows. The architecture adopts a service-oriented design, integrating distributed data management with dynamic metadata mapping to enable logical abstraction, on-demand loading, and unified access to heterogeneous intermediate data. Compared to conventional materialized storage approaches, it reduces storage overhead by an average of 62% (measured empirically) and substantially decreases inter-team collaboration latency. The system has been deployed in production, stably supporting six ML applications and over thirty concurrent workflows, demonstrating linear scalability. This work establishes a lightweight, elastic, and reusable data virtualization paradigm for large-scale ML infrastructure.

This work addresses the challenge of meeting deadlines while minimizing costs for concurrent streaming queries in cluster environments under dynamic workloads and unpredictable query arrivals. The paper proposes an intermittent query scheduling framework tailored for elastic parallel execution, which, to the best of our knowledge, is the first to integrate elastic resource provisioning into intermittent query processing. By dynamically scaling cluster nodes up or down, the approach simultaneously satisfies windowed query deadlines and reduces resource expenditure. Implemented on Apache Spark and deployed on AWS EMR, the system combines elastic computing with batch-oriented scheduling algorithms. Experimental evaluations on TPC-H and Yahoo Streaming Benchmark datasets demonstrate significant improvements over both static configurations and Spark Streaming, achieving superior timeliness and cost efficiency.

This work proposes OceanBase Mercury, a distributed near real-time analytical processing system built upon OceanBase, designed to deliver enterprise-grade analytical capabilities—including multi-tenancy, high availability, and elastic scalability—for petabyte-scale data. Traditional OLAP systems struggle to simultaneously support real-time transactions and efficient analytics at scale, often suffering from high data redundancy, complex cross-system synchronization, and poor timeliness. Mercury addresses these challenges through three key innovations: an adaptive columnar storage format with hybrid layout optimization, a materialized view differential refresh mechanism that ensures temporal consistency, and a polymorphic vectorized execution engine compatible with three distinct data formats. Experimental results on real-world workloads demonstrate that Mercury achieves 1.3–3.1× faster query latency than specialized OLAP engines while maintaining sub-second response times, effectively balancing analytical depth with operational agility.

This work addresses the limitations of traditional workflow platforms, which rely on static, pre-defined processes and struggle to accommodate the dynamic data integration demands of distributed systems. To overcome this, the authors propose a configuration-driven runtime orchestration framework that dynamically constructs execution graphs at request time through dependency-aware scheduling and parallel task execution, thereby circumventing the constraints of fixed workflows. This approach enables rapid adaptation to evolving integration scenarios without requiring system redeployment, significantly reducing latency. Empirical evaluation in a real-world Customer 360 enterprise use case demonstrates that the framework offers substantial advantages in flexibility, scalability, and efficient data aggregation compared to conventional solutions.

This work addresses the lack of systematic, reproducible, and maintainable testing methodologies in existing dynamic resource management libraries. We propose an automated validation framework tailored for high-performance computing (HPC) environments, which introduces a novel multi-level testing taxonomy encompassing both functional and non-functional requirements. Built upon an MPI-based scalable library testing methodology, the framework supports core primitives of dynamic resource management systems—such as initialization, readiness checks, and reconfiguration—and integrates containerized virtual clusters with continuous integration (CI) ecosystems. Experimental evaluation demonstrates that our approach significantly improves early fault detection rates, reduces maintenance overhead caused by evolving dependencies, and is readily generalizable to other systems exhibiting similar variability mechanisms.

This work addresses the inefficiency of existing large language model–based multi-agent systems in breadth-oriented search tasks, which struggle to meet the demands of large-scale, breadth-first information retrieval. Inspired by the MapReduce paradigm, the study introduces a novel horizontal parallel retrieval mechanism into multi-agent wide search, enabling efficient parallel processing through task-adaptive decomposition and result aggregation. Furthermore, an experience memory module is integrated to dynamically optimize query-driven task allocation and recombination. This approach overcomes the limitations of conventional vertical reasoning frameworks, achieving state-of-the-art performance across five benchmarks with Item F1 improvements of 5.11%–17.50% and a 45.8% reduction in runtime, significantly outperforming existing methods while offering superior cost-effectiveness.