dask

Prior work lacks systematic evaluation of how data processing frameworks impact end-to-end deep learning training and inference—particularly regarding performance-energy trade-offs across data loading, preprocessing, and batch feeding stages in conjunction with GPU computation. Method: We conduct the first comprehensive empirical study comparing Pandas, Polars, and Dask across diverse deep learning workloads—including CNNs and Transformers trained on ImageNet and WikiText—measuring runtime, memory footprint, disk I/O, and CPU/GPU power consumption under varying data scales and I/O characteristics. Contribution/Results: Polars achieves optimal latency–energy efficiency for medium-scale in-memory datasets; Dask scales effectively to ultra-large distributed workloads but exhibits lower energy efficiency; Pandas remains practical for small-batch, interactive tasks. Our findings bridge a critical gap in co-optimizing data engineering infrastructure with AI training pipelines, providing empirical guidance for green AI system design and framework selection in production ML systems.

Pandas struggles with ultra-large-scale data due to memory constraints, and migrating to scalable frameworks (e.g., Dask, Modin) requires extensive code rewriting. Method: This paper introduces the “Lazy Fat Panda Diet” paradigm—a JIT-based static analysis and lazy-loading DataFrame wrapper that enables seamless, API-compatible, backend-agnostic optimization across Pandas, Dask, and Modin—without modifying user code. The system supports lazy execution and runtime memory-aware query planning via just two lines of code. Contribution/Results: Evaluated on real-world data science workloads, our approach achieves 2.1–5.8× end-to-end speedup over baseline Pandas, maintains correctness and stability even under memory pressure, and eliminates costly framework migration. It significantly reduces development and deployment overhead for large-scale analytics while preserving full Pandas compatibility.

Existing Pandas APIs lack a dedicated benchmark, hindering systematic evaluation of performance and API coverage. Method: We introduce PandasBench—the first specialized benchmark suite for Pandas—built from 102 real-world Jupyter notebooks (3,721 code cells) and featuring four novel mechanisms: real-code-driven API coverage analysis, noise-aware code cleaning, multi-backend compatibility adaptation, and fine-grained, non-uniform input scaling—a first in Pandas benchmarking. Contribution/Results: Leveraging PandasBench, we conduct the largest empirical evaluation to date across four acceleration libraries—Modin, Dask, Koalas, and Dias. Results show only Modin achieves speedup in 8% of notebooks; Dias attains the highest speedup (54%) but introduces semantic errors; significant disparities exist across systems in both API support and correctness. PandasBench establishes a standardized infrastructure for rigorous performance assessment and optimization in the Pandas ecosystem.

This study systematically evaluates performance differences among Java, Python, and Scala for end-to-end ETL workloads on Apache Spark integrated with Apache Iceberg. We conduct controlled experiments across varying data scales (5 MB–1.6 GB) and operation complexities (basic transformations vs. complex merge operations), all within a uniform CSV→Spark→Iceberg pipeline. To our knowledge, this is the first standardized ETL benchmark enabling direct cross-language comparison under native Iceberg support. Results reveal nonlinear interactions among programming language, data volume, and operation type: Python exhibits superior throughput on small-scale data; performance converges across all three languages at medium scale (1.6 GB); and Scala significantly outperforms both Java and Python in high-complexity merge-intensive workloads. The findings provide empirical guidance for language selection in big-data ETL systems, along with a principled trade-off framework grounded in workload characteristics.

To address the challenge in cloud environments where HPC applications struggle to simultaneously achieve high resource utilization and low response latency for high-priority jobs—given that traditional MPI lacks automatic elasticity and existing elastic frameworks (e.g., Charm++) require manual application refactoring—this paper proposes the first Kubernetes Operator-based elastic HPC job scheduler. It innovatively integrates Charm++’s migratable object model with Kubernetes’ native orchestration capabilities, enabling fully automated, application-transparent dynamic scaling without source-code modifications. The scheduler employs a priority-aware elasticity algorithm to adjust resource allocation in real time within containerized environments. Experimental results demonstrate negligible scaling overhead, a 32% increase in average cluster resource utilization, and a 47% reduction in average response time for high-priority jobs—significantly outperforming static scheduling baselines.

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

This work addresses the inefficiency and insufficient automation in multi-objective optimization over high-dimensional parameter spaces commonly encountered in high-energy physics detector design. To tackle this challenge, the study presents the first deep integration of multi-objective Bayesian optimization with the PanDA-iDDS distributed workflow system, establishing a general-purpose and scalable intelligent optimization framework. By orchestrating collaborative simulation tasks across heterogeneous computing resources, the framework substantially enhances the automation, computational efficiency, and scalability of the optimization process. Its effectiveness is demonstrated through successful applications in the design optimization of both ePIC and dRICH detectors.

This work addresses the high barrier faced by novice users in deploying and optimizing big data workflows on high-performance computing (HPC) systems. To this end, it proposes an interactive JupyterHub-integrated plugin that, for the first time, unifies HPC-aware big data cluster lifecycle management and performance evaluation within a notebook environment. The approach seamlessly integrates resource scheduling, multi-cluster orchestration, performance metric collection, and visualization, enabling users to perform iterative optimization through an intuitive, one-click interface. Experimental results demonstrate that users can accomplish application deployment, performance assessment, and tuning with only a few clicks, substantially enhancing both usability and execution efficiency of big data workflows on HPC platforms.

Scientific computing in Python is constrained by single-node parallelism, hindering efficient migration of NumPy-based prototypes to multi-GPU supercomputing clusters; meanwhile, traditional HPC abstractions lack resource elasticity and usability. To address this, we propose CharmTyles—a Charm++-based adaptive stencil computation framework supporting multi-node GPU execution. It introduces the first stencil abstraction that unifies NumPy-like syntax with runtime cross-node dynamic reconfiguration. Leveraging distributed memory management, automatic data partitioning, and optimized asynchronous communication, CharmTyles achieves elastic resource scaling with sub-200 ms reconfiguration latency. Evaluation shows significant performance improvements over both domain-specific stencil DSLs and general-purpose NumPy alternatives. CharmTyles effectively bridges the critical gap between rapid scientific prototyping and scalable deployment on exascale-class systems.

This work addresses the challenges of poor scalability and accuracy degradation in scientific machine learning when handling extremely high-resolution data, particularly due to the absence of a general-purpose parallelization framework supporting sub-unit batch sizes per device. The authors propose ShardTensor, a novel domain-parallel paradigm that shards tensors along spatial domains, thereby decoupling data dimensions from hardware constraints. This approach enables, for the first time, general-purpose parallel training and inference with sub-unit batch sizes. By supporting multidimensional parallelism and integrating dynamic computation–communication load balancing, ShardTensor simultaneously achieves strong scaling—reducing latency—and weak scaling—enabling larger-scale data processing—thereby significantly enhancing the scalability and efficiency of high-fidelity scientific computing tasks.