apache spark

To address the joint optimization challenges of performance, maintainability, and collaborative efficiency in large-scale integrated machine learning within distributed data processing systems, this paper proposes Pipes—a declarative, modular data pipeline architecture. Pipes decomposes pipelines into logically encapsulated computation units, implemented atop Apache Spark with standardized interfaces and well-defined component boundaries—departing from conventional microservice paradigms to enable high-performance, maintainable ML pipeline development. In enterprise deployments, Pipes improves development efficiency by 50%, reduces collaborative debugging cycles from weeks to days, achieves 500× scalability, and delivers 10× higher throughput. Academic benchmarks show >5.7× throughput improvement and 99% CPU utilization. Its core contribution is the first deep integration of declarative abstractions with Spark’s native execution model, simultaneously advancing both development methodology and system performance.

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

This study systematically evaluates performance differences among Java, Python, and Scala for end-to-end ETL workloads on Apache Spark integrated with Apache Iceberg. We conduct controlled experiments across varying data scales (5 MB–1.6 GB) and operation complexities (basic transformations vs. complex merge operations), all within a uniform CSV→Spark→Iceberg pipeline. To our knowledge, this is the first standardized ETL benchmark enabling direct cross-language comparison under native Iceberg support. Results reveal nonlinear interactions among programming language, data volume, and operation type: Python exhibits superior throughput on small-scale data; performance converges across all three languages at medium scale (1.6 GB); and Scala significantly outperforms both Java and Python in high-complexity merge-intensive workloads. The findings provide empirical guidance for language selection in big-data ETL systems, along with a principled trade-off framework grounded in workload characteristics.

To address the challenge of processing multi-source, heterogeneous data in social networks, this paper proposes a unified batch-stream-graph analytics framework built upon the Hadoop-Spark ecosystem. The method systematically integrates Hive (for SQL-based batch processing), HBase (for low-latency key-value lookups), and GraphX (for scalable graph computation) under a single Spark execution layer. It supports three core analytical tasks: user influence assessment, high-frequency term statistics, and community relationship mining. Leveraging HDFS for distributed storage, YARN for resource orchestration, and multi-language APIs, the framework achieves loosely coupled integration of computation and storage. End-to-end experiments on real-world social datasets demonstrate that the hybrid architecture accelerates complex relational analysis by 1.8–3.2× compared to single-component baselines, significantly improving both processing efficiency and system flexibility.

This study addresses the high learning barrier and conceptual abstraction inherent in AI–big data interdisciplinary education. Methodologically, it proposes a pedagogical paradigm integrating *conceptual simplification*, *intuitive visualization*, and *full-stack technical integration*. It systematically unifies core deep learning architectures (CNNs, ResNet, YOLO, Transformers), pre-trained models (BERT, GPT), and big data technologies (SQL/NoSQL, Hadoop, Spark), delivering knowledge coherence through principled explanations, dynamic visualizations, and cross-modal case studies (NLP, image recognition, autonomous driving). Its key contribution is the first unified teaching framework spanning neural network fundamentals, transfer learning with pre-trained models, and big data–enabled AI deployment. Empirical evaluation demonstrates that the paradigm significantly accelerates beginner onboarding, improves downstream task accuracy by 15–30%, and reduces AI application development cycles by over 40%.

This work addresses the lack of a universal, flexible, and cluster-agnostic workload representation in existing distributed machine learning systems, which hinders efficient design space exploration. To overcome this limitation, the paper introduces Flint, a novel framework that leverages the intermediate representation of machine learning compilers to extract workload graphs for clusters of arbitrary scale—without requiring actual hardware execution. By decoupling workload modeling from underlying hardware specifics and validating accuracy through execution traces, Flint ensures both fidelity and portability. Experimental results demonstrate that Flint effectively enables flexible and efficient design space exploration while substantially reducing evaluation overhead.

Prior work lacks systematic evaluation of how data processing frameworks impact end-to-end deep learning training and inference—particularly regarding performance-energy trade-offs across data loading, preprocessing, and batch feeding stages in conjunction with GPU computation. Method: We conduct the first comprehensive empirical study comparing Pandas, Polars, and Dask across diverse deep learning workloads—including CNNs and Transformers trained on ImageNet and WikiText—measuring runtime, memory footprint, disk I/O, and CPU/GPU power consumption under varying data scales and I/O characteristics. Contribution/Results: Polars achieves optimal latency–energy efficiency for medium-scale in-memory datasets; Dask scales effectively to ultra-large distributed workloads but exhibits lower energy efficiency; Pandas remains practical for small-batch, interactive tasks. Our findings bridge a critical gap in co-optimizing data engineering infrastructure with AI training pipelines, providing empirical guidance for green AI system design and framework selection in production ML systems.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

To address the high communication overhead and poor scalability of serverless architectures in machine learning–intensive, data-heavy workloads, this paper proposes a high-performance computing (HPC)-inspired serverless framework. Methodologically, it introduces a NAT-traversal direct communication mechanism based on TCP hole punching, implements a lightweight serverless communicator, and integrates the Cylon distributed dataframe library with an FMI-inspired heuristic communication scheduling model. This design enables decentralized, low-latency, high-throughput distributed data processing within cloud-native environments—without centralized coordination. Experimental results demonstrate that the framework achieves over 99% end-to-end performance improvement compared to conventional serverless approaches. Its strong scaling efficiency closely matches that of EC2 instances and dedicated HPC clusters. Notably, this work is the first to achieve near-HPC communication efficiency and scalability in a serverless setting.

This work addresses the challenge of Spark SQL performance tuning, which is hindered by an enormous configuration space and the prohibitive cost of full-fidelity evaluation, making it difficult to identify high-quality configurations within practical time limits. To overcome this, the authors propose MFTune, a novel approach that introduces a query-level multi-fidelity partitioning mechanism to construct a representative, low-cost performance surrogate using a carefully selected subset of SQL queries with high correlation to overall workload behavior. MFTune further integrates density-based clustering for parameter space compression, tailored transfer learning, and a two-stage warm-start strategy to dramatically enhance tuning efficiency. Experimental results on TPC-H and TPC-DS benchmarks demonstrate that MFTune consistently outperforms five state-of-the-art methods under realistic time constraints, rapidly converging to superior configurations.