hadoop

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

In existing distributed stream processing frameworks, data processing logic is deeply coupled with distribution strategies, limiting scheduling flexibility in high-level frameworks and imposing excessive development overhead on low-level ones. Method: This paper proposes a pluggable stream processing framework that achieves full decoupling between processing logic and distribution strategies for the first time. It supports modular definition, reuse, and composition of distribution strategies via a domain-specific language (DSL). The framework integrates a runtime strategy plugin mechanism, streaming topology compilation optimizations, and a lightweight Storm-compatible execution engine. Contribution/Results: Experiments show that the framework achieves throughput comparable to Apache Storm, successfully reproduces multiple state-of-the-art distribution strategies, and validates the correctness, predictable performance, and engineering feasibility of modular strategy design—thereby bridging the critical gap between high-level abstraction expressiveness and low-level scheduling controllability.

Distributed file systems (DFS) exhibit divergent fault tolerance and horizontal scalability characteristics, yet systematic, cross-architecture evaluation under realistic hybrid workloads remains scarce. Method: We conduct a comprehensive empirical study of Google File System (GFS), Hadoop Distributed File System (HDFS), and MinIO—representing legacy, big-data, and cloud-native paradigms—using unified benchmarks on physical clusters. We apply stress testing, controlled fault injection, and protocol-level log analysis to quantify throughput, recovery latency, and consistency guarantees across data redundancy, node failure recovery, and high-concurrency client access. Contribution/Results: This is the first study to comparatively evaluate these three architecturally distinct DFS under concurrent cloud-native and big-data workloads. We identify MinIO’s low-latency advantage for small files, HDFS’s stability limits in batch processing, and GFS’s enduring influence on lightweight DFS design. Based on these findings, we propose a scenario-driven selection framework—distinguishing high-availability storage from analytical workloads—to guide industrial storage system deployment with empirical evidence.

Managed big-data frameworks (e.g., Spark, Giraph) suffer from high garbage collection (GC) overhead due to Java heap memory pressure, while offloading objects to external storage incurs substantial serialization/deserialization (S/D) costs; merely scaling DRAM exacerbates low CPU core utilization. Method: We identify the synergistic bottleneck of GC and I/O overheads on server throughput and propose TeraHeap—a two-tier heap mechanism enabling dynamic DRAM budget allocation between the Java heap (H1) and page cache (PC), jointly optimized via JVM tuning and multi-instance memory co-location. Contribution/Results: Across diverse memory–core configurations, TeraHeap improves effective CPU utilization by up to 2.3× over baseline approaches, significantly boosting server throughput. It establishes a lightweight, system-level memory coordination paradigm for resource-constrained big-data execution environments.

Scheduling data-intensive workloads in large-scale distributed systems faces challenges including complexity, heterogeneous parallelism, data locality constraints, and multi-dimensional QoS optimization (e.g., timeliness, fault tolerance, energy efficiency). Method: This paper proposes a novel workload classification scheme grounded in data characteristics and service requirements; systematically surveys and structures mainstream scheduling strategies, exposing critical limitations in dynamic adaptability, fine-grained fault tolerance, and energy–QoS co-optimization; and introduces a unified scheduling framework integrating data-locality awareness, elastic parallel scheduling, QoS-tiered guarantees, and energy-aware resource allocation. Contribution/Results: The study establishes a scalable classification paradigm, delivers a clear technology evolution roadmap, and identifies a prioritized list of open research challenges—thereby advancing foundational understanding and guiding future design of intelligent, holistic schedulers for modern distributed data systems.

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

This work addresses the performance degradation of large-scale HPC applications in shared storage systems caused by imbalanced I/O bandwidth allocation, where small jobs often monopolize resources at the expense of larger ones. To resolve this, the authors propose AdapTBF, a decentralized, adaptive bandwidth regulation scheme based on a token bucket mechanism, designed for integration into parallel file systems such as Lustre. Unlike conventional static rate-limiting approaches, AdapTBF dynamically enables idle bandwidth borrowing and lending to accommodate bursty I/O demands while preserving fairness and storage efficiency. Experimental results under real-world workloads demonstrate that AdapTBF significantly improves aggregate throughput, effectively safeguards the performance of large jobs, and maintains high resource utilization alongside equitable bandwidth distribution even in extreme scenarios.

This work addresses the challenges of high energy consumption and inefficient resource scheduling in cloud data centers under big data workloads. The authors propose an adaptive resource scheduling mechanism based on workload profiling, which integrates historical logs and real-time telemetry data to model CPU, memory, and storage I/O characteristics. This model predicts the energy and performance implications of virtual machine placement decisions and dynamically triggers resource consolidation. Evaluated on a multi-node cloud platform using representative workloads—including Hadoop MapReduce, Spark MLlib, and ETL pipelines—the approach achieves consistent energy savings of 15%–20% over baseline schedulers while incurring negligible performance degradation, all while adhering to service-level agreements. The method thus effectively balances energy efficiency and sustainability without compromising operational performance.

To address insufficient multi-objective load balancing in stream processing systems under complex workloads, this paper proposes a multi-tier collaborative scheduling framework. The framework introduces dynamic inter-layer coordination mechanisms and lightweight interfaces among schedulers, enabling seamless integration of novel scheduling policies. It jointly optimizes computational resource utilization, end-to-end latency, and throughput by integrating multi-objective optimization, distributed resource management, and real-time feedback control. Its key innovation lies in shifting hierarchical scheduling from static decoupling to dynamic collaboration—preserving scalability while significantly enhancing adaptability. Evaluated in Meta’s production environment, the system reliably processes TB-scale data with sub-second latency; it improves critical resource utilization by 27% and reduces tail latency by 41%.

This work addresses the challenges of inconsistent interfaces, limited flexibility, and poor integration with resource management systems in dynamic resource management for high-performance computing. To overcome these issues, the authors propose a modular Dynamic Resource Management (DMR) framework that abstracts underlying heterogeneity through a unified API. The framework integrates the Proteo reconfiguration engine with MPI Spawn to support diverse runtime reconfiguration strategies, thereby eliminating the need for full-process restarts while maintaining compatibility with mainstream resource managers. Experimental evaluation on the MPDATA solver demonstrates that the proposed approach significantly enhances both programming productivity and runtime performance, enabling efficient and scalable dynamic resource scheduling and reconfiguration.