performance engineering

Modern software systems frequently exhibit application-level resource contention bottlenecks—such as blocking on custom application events—that evade detection by conventional performance profilers due to complex dependencies and bespoke resource management. To address this, we propose OmniResource Profiling, the first method to jointly leverage system-level metrics and application-level event-waiting relationships. It employs a lightweight LLM-assisted static analysis to automatically identify custom resources and cross-execution-trace runtime variable comparison for precise root-cause localization. Evaluated on 12 known performance issues across five real-world applications, OmniResource achieves 100% diagnostic accuracy and uncovers two previously undetected bottlenecks. Crucially, it requires no intrusive instrumentation, balancing high precision with practical deployability. This work delivers the first end-to-end solution for application-level resource contention analysis.

To address the challenges of late detection of performance degradation and delayed scalability analysis in HPC application development, this paper proposes a lightweight, CI-native continuous performance monitoring framework. The method integrates TALP-based real-time performance instrumentation with in-repository regression analysis, enabling low-overhead (no additional tracing) and high-temporal-fidelity monitoring—delivering feedback immediately upon CI build completion. Performance data are stored in a CI-friendly directory structure, and automated HTML reports are generated, visualizing trends in key performance factors as well as strong and weak scaling efficiency. Evaluated in the GENE-X CI environment with zero code modification, the framework demonstrates sensitivity to minute performance improvements (<2%) and reduces post-processing overhead by over 90% compared to conventional tracing tools. This significantly enhances the efficiency and practicality of scalability assessment under resource-constrained conditions.

Existing benchmarks for execution-time optimization patches primarily target Python, C++, or .NET, lacking configurable, reproducible solutions tailored to Java. This work proposes JETO-Mine, the first framework for automatically mining and validating Java performance patches with customizable filtering and statistically rigorous validation. JETO-Mine employs a three-stage pipeline integrating static analysis, LLM-driven issue categorization, Docker-based dynamic testing, and significance testing to construct JETO-Bench—a benchmark comprising 660 candidate patches and 91 manually verified effective ones. Experimental evaluation demonstrates that JETO-Bench effectively assesses patch generation tools (e.g., OpenHands achieves a 14.3% repair success rate) and reveals a widespread absence of performance validation tests in Java projects.

This work addresses the high overhead of processing massive telemetry data in exascale supercomputing systems by proposing a heterogeneous acceleration–enabled, high-performance diagnostic framework. Integrating high-throughput C++ APIs with GPU-parallelized computation, the framework supports scalable MPI trace analysis and seamless integration with external tools. It introduces a novel topology-aware workflow that maps logical performance anomalies onto the physical coordinates of the Slingshot interconnect and pioneers a three-dimensional performance model to iteratively reconstruct application behavior, enabling precise identification of performance headroom. Evaluated on Aurora, the system ingests traces from 100,000 MPI ranks in just 9.69 seconds, achieving up to a 314× speedup over CPU-based analysis. On Frontier, it uncovers 32.28% potential acceleration for the GAMESS application.

Existing evaluations of code-generating agents primarily emphasize functional correctness while overlooking their capacity for performance optimization in real-world scenarios. This work proposes the first end-to-end benchmark tailored to the full performance engineering lifecycle, requiring agents to achieve reproducible performance gains through profiling, diagnosis, code modification, and validation—all while preserving functional correctness. The framework introduces several innovations, including hidden correctness tests, verifiable speedup metrics, and trajectory auditing, integrated with program profiling, cross-layer bottleneck diagnosis, large model–agent collaboration, and an optimization-summary handoff strategy. Evaluation across seven long-horizon tasks and seven agent stacks reveals that optimization efficacy is highly workload-dependent, with no single dominant approach; relying solely on raw speedup ratios can lead to misleading conclusions, necessitating a holistic assessment that jointly considers correctness and reproducibility.

This work addresses the computational and memory bottlenecks that hinder efficient scaling in large model training. To overcome the limitations of conventional point-wise optimizations, the authors propose a throughput-centric strategy that systematically integrates multiple techniques: optimized data loading (OVERLORD), CPU memory offloading (DeepSpeed ZeRO-Offload), distributed compilation (Triton-distributed), and hardware-level dynamic voltage and frequency scaling (DVFS). This holistic approach achieves a 4.5% improvement in end-to-end training throughput, substantially reduces training costs, and enables efficient training of models significantly larger than the memory capacity of a single GPU.

Traditional efficiency metrics struggle to accurately assess resource utilization in heterogeneous high-performance computing systems that combine CPUs and accelerators. This work extends the POP efficiency model by introducing a hardware-agnostic, host-device dual-branch hierarchical efficiency framework. It uniquely defines a multiplicative efficiency decomposition on the device side, symmetric to that on the host, separately capturing mixed execution/offload efficiency and device parallel efficiency. Implemented via the lightweight TALP monitoring library, the approach supports both runtime and post-mortem analysis and outputs results in human-readable and machine-readable formats. Experiments on synthetic benchmarks and three real-world HPC applications demonstrate that the proposed methodology effectively uncovers performance bottlenecks related to offloading, load balancing, and task scheduling, offering developers actionable insights for optimization.

This study addresses the limited sensitivity of traditional cloud service performance regression detection, which is often hindered by I/O fluctuations and infrastructure changes. The authors propose a novel paradigm termed “Duet Instrumentation,” which uniquely integrates large language model (LLM)-driven code change analysis with synchronized dual-version benchmarking. By leveraging an LLM to precisely identify performance-relevant changes between consecutive versions, the method dynamically instruments only those critical code regions, achieving high-sensitivity regression detection with low overhead. Evaluated in real-world environments, the approach attains a precision of 58%, recall of 93%, and specificity of 71%, effectively detecting performance regressions as subtle as one-fifth the severity detectable by conventional methods.

This work addresses the challenge of efficiently conducting “What-If” I/O performance analysis for large-scale HPC applications, which is hindered by the complex interplay among access patterns, middleware, and file systems. The authors propose FBench, the first flexible I/O benchmarking tool based on context-free grammars (CFGs), capable of generating or replaying I/O traces—captured via Recorder—in real time without modifying application code. FBench supports both POSIX and MPI-IO interfaces and enables configuration-driven exploration through JSON-defined optimization strategies. It faithfully reproduces real-world workloads such as IOR, HACC-IO, FLASH Sedov, and LAMMPS. Evaluations on Lustre reveal that collective I/O write bandwidth can be up to 30× lower than ideal, burst buffers improve non-collective write bandwidth by 1.5×, and performance gains of up to 8× are achievable in LAMMPS scenarios, significantly accelerating I/O optimization studies.