Using a Sledgehammer to Crack a Nut? Revisiting Automated Compiler Fault Isolation

2025-12-18

AI Summary

While developers commonly identify bug-introducing commits (BICs) via binary search over version histories to localize compiler bugs, existing spectrum-based fault localization (SBFL) techniques have not been systematically benchmarked against this simple yet widely adopted strategy. Method: This paper introduces the BIC-localization strategy (“Basic”) as a baseline and conducts a comparative evaluation against state-of-the-art SBFL methods on 60 real-world defects each in GCC and LLVM. BICs are localized via binary search guided by version history analysis and SBFL-based bug reproduction; Top-1 and Top-5 localization accuracy are quantified. Contribution/Results: Basic matches or surpasses advanced SBFL techniques across most scenarios, exposing critical limitations in current SBFL evaluation paradigms. The study establishes BIC localization as a new practical benchmark for compiler defect localization and provides methodological insights for evaluating the real-world utility of fault localization techniques.

Technology Category

• Search and Optimization: Local Search
• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Background: Compilers are fundamental to software development, translating high-level source code into executable software systems. Faults in compilers can have severe consequences and thus effective localization and resolution of compiler bugs are crucial. Problem: In practice, developers often examine version history to identify and investigate bug-inducing commit (BIC) for fixing bugs. However, while numerous sophisticated Spectrum-Based Fault Localization (SBFL) techniques have been proposed for compiler fault isolation, their effectiveness has not been evaluated against the BIC-based strategies widely adopted in practice. Objective: This study aims to bridge this gap by directly comparing a BIC-based strategy, Basic, with representative SBFL techniques in the context of compiler fault localization. The BIC-based strategy closely aligns with common developer practices, as it directly identifies the BIC and treats the files modified in that commit as faulty candidates. Method: The Basic identifies the most recent good release and earliest bad release, and then employs a binary search to pinpoint the bug-inducing commit. All files modified in the identified commit are flagged as potentially faulty. We rigorously compare Basic against SBFL-based techniques using a benchmark consisting of 60 GCC bugs and 60 LLVM bugs. Result: Our analysis reveals that Basic performs comparably to, and in many cases outperforms, state-of-the-art SBFL-based techniques, particularly on the critical Top-1 and Top-5 ranking metrics. Conclusion: This study provides new insights into the practical effectiveness of SBFL-based techniques in real-world compiler debugging scenarios. We recommend that future research adopt Basic as a baseline when developing and evaluating new compiler fault isolation methods.

Problem

• Evaluates BIC-based vs SBFL techniques for compiler bug localization
• Compares practical developer strategies with automated fault isolation methods
• Assesses effectiveness on real GCC and LLVM bugs using ranking metrics

Innovation

• Basic strategy uses binary search to identify bug-inducing commits
• Compares BIC-based approach with Spectrum-Based Fault Localization techniques
• Flags all files modified in bug-inducing commit as faulty candidates