Similar Pattern Annotation via Retrieval Knowledge for LLM-Based Test Code Fault Localization

2026-05-08

AI Summary

Test Code Fault Localization (TCFL) in continuous integration is hindered by black-box environments, sparse diagnostic signals, and vast search spaces, limiting the effectiveness of existing approaches. This work proposes SPARK, a novel framework that synergistically integrates historical debugging knowledge with large language models (LLMs). SPARK employs a retrieval-augmented mechanism to identify historically similar failing test cases from CI logs and applies selective line-level annotations to guide the LLM toward suspicious code regions. This strategy avoids prompt length explosion while significantly improving localization accuracy, particularly in complex scenarios involving multiple faults. Experimental results demonstrate that SPARK outperforms current LLM-based TCFL methods on three industrial-scale Python test datasets, achieving higher precision in identifying multiple fault locations while maintaining manageable inference overhead and token consumption.

Abstract

Software failures remain a major challenge in modern software development, and identifying the code elements responsible for failures is a time-consuming debugging task. While extensive research has focused on fault localization in the system under test (SUT), failures can also originate from faulty system test scripts. This problem, known as Test Code Fault Localization (TCFL), has received significantly less attention despite its importance in continuous integration (CI) environments where large test suites are executed frequently. TCFL is particularly challenging because it typically operates under black-box conditions, relies on limited diagnostic signals such as error messages and partial logs, and involves large system-level test scripts that expand the fault localization search space. In this paper, we propose SPARK, a framework that integrates accumulated debugging knowledge from continuous integration (CI) environments into Large Language Model (LLM)-based TCFL. Given a newly observed failing test case, SPARK retrieves similar fault-labeled test cases from a debugging knowledge corpus and selectively annotates suspicious lines of the failing test based on their similarity to previously observed fault patterns. These annotations guide the LLM's reasoning while maintaining scalability and avoiding the prompt-length explosion common to naive retrieval-augmented approaches. We evaluate SPARK on three industrial datasets containing real-world faulty Python test cases from different software products. The results show that SPARK consistently improves fault localization effectiveness compared to the existing LLM-based TCFL baseline while maintaining comparable inference cost and token usage. In particular, the approach advances the state of the art by identifying more correct faulty locations in complex test cases containing multiple faults.

Problem

• Test Code Fault Localization
• Continuous Integration
• Fault Localization
• Debugging
• Large Language Models

Innovation

• Test Code Fault Localization
• Large Language Model
• Retrieval-Augmented Debugging
• Continuous Integration
• Similarity-Based Annotation