Detecting Performance-Relevant Changes in Configurable Software Systems

2025-11-21

AI Summary

Detecting performance regressions in configurable software is costly, and configuration sampling often misses localized performance degradation. Method: This paper proposes ConfFLARE, a technique that combines data-flow dependency analysis with change-impact propagation tracking to identify code changes interacting—via data flow—with performance-sensitive code. It further integrates configuration-feature identification to automatically select the subset of performance-sensitive configurations most likely affected by each change. Contribution/Results: ConfFLARE eliminates the need for exhaustive configuration-based performance testing. In evaluations on synthetic and real-world systems, it reduces the number of required test configurations by 79% and 70%, respectively, while achieving near-complete coverage of performance regression cases. It precisely pinpoints relevant features and significantly improves both the efficiency and completeness of performance regression detection.

Technology Category

• Search and Optimization: Algorithm Configuration
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Machine Learning: Evaluation and Analysis

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Performance is a volatile property of a software system and frequent performance profiling is required to keep the knowledge about a software system's performance behavior up to date. Repeating all performance measurements after every revision is a cost-intensive task, especially in the presence of configurability, where one has to measure multiple configurations to obtain a comprehensive picture. Configuration sampling is a common approach to control the measurement cost. However, it cannot guarantee completeness and might miss performance regressions, especially if they only affect few configurations. As an alternative to solve the cost reduction problem, we present ConfFLARE: ConfFLARE estimates whether a change potentially impacts performance by identifying data-flow interactions with performance-relevant code and extracts which software features participate in such interactions. Based on these features, we can select a subset of relevant configurations to focus performance profiling efforts on. In a study conducted on both, synthetic and real-world software systems, ConfFLARE correctly detects performance regressions in almost all cases and identifies relevant features in all but two cases, reducing the number of configurations to be tested on average by $79%$ for synthetic and by $70%$ for real-world regression scenarios saving hours of performance testing time.

Problem

• Detecting performance-relevant changes in configurable software systems efficiently
• Reducing measurement costs by selecting relevant configurations for testing
• Identifying performance regressions through data-flow interactions with critical code

Innovation

• Identifies data-flow interactions with performance-relevant code
• Extracts software features involved in performance interactions
• Selects relevant configurations to reduce testing efforts