big data

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

This study systematically evaluates performance differences among Java, Python, and Scala for end-to-end ETL workloads on Apache Spark integrated with Apache Iceberg. We conduct controlled experiments across varying data scales (5 MB–1.6 GB) and operation complexities (basic transformations vs. complex merge operations), all within a uniform CSV→Spark→Iceberg pipeline. To our knowledge, this is the first standardized ETL benchmark enabling direct cross-language comparison under native Iceberg support. Results reveal nonlinear interactions among programming language, data volume, and operation type: Python exhibits superior throughput on small-scale data; performance converges across all three languages at medium scale (1.6 GB); and Scala significantly outperforms both Java and Python in high-complexity merge-intensive workloads. The findings provide empirical guidance for language selection in big-data ETL systems, along with a principled trade-off framework grounded in workload characteristics.

To address systemic inefficiencies—including resource idleness, redundant data transfers, and violations of data locality—arising from misaligned coordination between the PanDA workflow system and the Rucio data management system in the ATLAS experiment, this paper proposes an end-to-end co-optimization framework. We introduce a novel file-level metadata matching algorithm to precisely associate computing tasks with datasets, and integrate log-based tracing, spatiotemporal imbalance analysis, and anomaly pattern detection to construct a fine-grained, holistic view of data access and movement. Our approach is the first to identify, in production, the root causes of cross-system scheduling mismatches, delivering interpretable performance insights. Empirical validation confirms tangible improvements in resource utilization and system resilience, demonstrating the feasibility and effectiveness of the proposed co-design strategies.

To address the joint optimization challenges of performance, maintainability, and collaborative efficiency in large-scale integrated machine learning within distributed data processing systems, this paper proposes Pipes—a declarative, modular data pipeline architecture. Pipes decomposes pipelines into logically encapsulated computation units, implemented atop Apache Spark with standardized interfaces and well-defined component boundaries—departing from conventional microservice paradigms to enable high-performance, maintainable ML pipeline development. In enterprise deployments, Pipes improves development efficiency by 50%, reduces collaborative debugging cycles from weeks to days, achieves 500× scalability, and delivers 10× higher throughput. Academic benchmarks show >5.7× throughput improvement and 99% CPU utilization. Its core contribution is the first deep integration of declarative abstractions with Spark’s native execution model, simultaneously advancing both development methodology and system performance.

To address the challenge of processing multi-source, heterogeneous data in social networks, this paper proposes a unified batch-stream-graph analytics framework built upon the Hadoop-Spark ecosystem. The method systematically integrates Hive (for SQL-based batch processing), HBase (for low-latency key-value lookups), and GraphX (for scalable graph computation) under a single Spark execution layer. It supports three core analytical tasks: user influence assessment, high-frequency term statistics, and community relationship mining. Leveraging HDFS for distributed storage, YARN for resource orchestration, and multi-language APIs, the framework achieves loosely coupled integration of computation and storage. End-to-end experiments on real-world social datasets demonstrate that the hybrid architecture accelerates complex relational analysis by 1.8–3.2× compared to single-component baselines, significantly improving both processing efficiency and system flexibility.

Scheduling data-intensive workloads in large-scale distributed systems faces challenges including complexity, heterogeneous parallelism, data locality constraints, and multi-dimensional QoS optimization (e.g., timeliness, fault tolerance, energy efficiency). Method: This paper proposes a novel workload classification scheme grounded in data characteristics and service requirements; systematically surveys and structures mainstream scheduling strategies, exposing critical limitations in dynamic adaptability, fine-grained fault tolerance, and energy–QoS co-optimization; and introduces a unified scheduling framework integrating data-locality awareness, elastic parallel scheduling, QoS-tiered guarantees, and energy-aware resource allocation. Contribution/Results: The study establishes a scalable classification paradigm, delivers a clear technology evolution roadmap, and identifies a prioritized list of open research challenges—thereby advancing foundational understanding and guiding future design of intelligent, holistic schedulers for modern distributed data systems.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

This study addresses the challenge of selecting an optimal Data Lakehouse architecture based on data type and scale by presenting the first systematic evaluation of Apache Hudi, Apache Iceberg, and Delta Lake in terms of data ingestion efficiency and storage overhead for structured and semi-structured workloads. Conducted on the Apache Spark platform, the empirical comparison employs a four-stage ETL pipeline to assess the three frameworks under realistic conditions. Experimental results demonstrate that Delta Lake achieves the fastest data loading performance, while Iceberg excels in storage compression ratio and system stability. In contrast, Hudi exhibits comparatively lower efficiency in both batch ingestion and storage utilization. These findings provide critical empirical evidence and practical guidance for informed architectural decisions in Lakehouse deployments.

Existing Computational Object Storage (COS) systems face three key bottlenecks when performing large-scale scientific tabular data SQL analytics in HPC environments: rigid output formats, limited operator pushdown capability, and inadequate adaptation to deep storage hierarchies. To address these, we propose COS-SQL—a near-data SQL analytics framework tailored for HPC. Our approach features: (1) flexible output format support—including Arrow columnar layout; (2) full-stage pushdown of complex operators and array expressions; and (3) dynamic execution path selection based on hierarchical storage structure. COS-SQL adopts object-level storage organization and tightly integrates with Apache Spark. Evaluated on real-world HPC workloads, it achieves up to 32.7% end-to-end performance improvement over state-of-the-art COS systems, significantly enhancing both analytical flexibility and execution efficiency.

Data engineering and AI/ML platforms face inherent trade-offs among performance, security, usability, and seamless integration with existing data architectures. Method: This paper proposes Snowpark—a platform built upon Snowflake’s elastic architecture—that (1) introduces a Python package caching mechanism to drastically reduce query initialization latency; (2) implements a customized workload scheduler with row-level redistribution to mitigate data skew; and (3) enforces tenant-level isolation via secure sandboxes, enabling robust multi-language (especially Python) support and deep control-plane integration. Results: Evaluated in production environments, the solution improves execution efficiency by 37%, increases resource utilization by 2.1×, and ensures strict tenant isolation and zero-trust security—establishing a scalable, secure, low-latency systems paradigm for cloud-native data intelligence platforms.