JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java

2026-06-30

AI Summary

Existing benchmarks for execution-time optimization patches primarily target Python, C++, or .NET, lacking configurable, reproducible solutions tailored to Java. This work proposes JETO-Mine, the first framework for automatically mining and validating Java performance patches with customizable filtering and statistically rigorous validation. JETO-Mine employs a three-stage pipeline integrating static analysis, LLM-driven issue categorization, Docker-based dynamic testing, and significance testing to construct JETO-Bench—a benchmark comprising 660 candidate patches and 91 manually verified effective ones. Experimental evaluation demonstrates that JETO-Bench effectively assesses patch generation tools (e.g., OpenHands achieves a 14.3% repair success rate) and reveals a widespread absence of performance validation tests in Java projects.

Abstract

Automated fixing of performance issues is gaining increasing attention. However, existing benchmarks of execution time improvement patches are fixed datasets that target Python, C++, or .NET and cannot be extended to new patches according to user-defined configurations. In this paper, we present JETO-Mine, the first configurable and reusable tool for automatically creating reproducible benchmarks of execution time improvement patches (ETIPs) in real-world Java projects. JETO-Mine employs a three-phase pipeline: a static analysis phase that crawls GitHub repositories and identifies ETIPs using user-defined filters and an LLM-based issue classifier, a dynamic analysis phase that wraps the identified ETIPs in Docker images for fully reproducible execution and performs statistical testing to find objective evidence of execution time improvement, and an evaluation harness that enables quantitative assessment of both generated patches and generated tests. Unlike existing benchmarks, JETO-Mine is designed as a reusable tool that allows researchers continuously collect new benchmarks with their own desired filters and statistical rigor levels. We use JETO-Mine to build JETO-Bench, a benchmark of 660 identified ETIPs and 91 manually verified executable ETIPs collected from 174 open-source Java repositories. To build JETO-Bench, JETO-Mine scans 11 years of open-source development history and nearly 1.8 million commits. We run OpenHands, a leading open-source coding agent, on the 91 manually verified executable ETIPs in JETO-Bench and find that it correctly fixes 14.3% (13/91) of the issues, aligning with results reported by similar studies on other programming languages. Our results also reveal that open-source Java projects largely lack tests that demonstrate execution time improvements, presenting an opportunity for future research in test generation.

Problem

• execution time improvement
• benchmark
• Java
• reproducibility
• performance bug

Innovation

• execution time improvement patches
• reproducible benchmark
• configurable mining tool
• static and dynamic analysis
• Java performance optimization