Fail-Aware and Explainable Test Oracle Prediction

2026-07-13

AI Summary

This work addresses the longstanding challenge of constructing effective test oracles, noting that existing learning-based assertion generation methods, while syntactically correct, exhibit limited fault-detection capability. The paper proposes FOCAL, a discriminative test oracle predictor based on large language models for code, which directly predicts whether a given test prefix passes or fails. FOCAL innovatively integrates failure-aware training with statement-level behavioral evidence to produce interpretable predictions. By employing a loss function specifically designed to emphasize failing test cases and incorporating explainability mechanisms grounded in program behavior, FOCAL significantly outperforms the SEER baseline on unseen projects. It not only achieves higher accuracy in fault prediction but also provides high-confidence explanations that can be validated through observable program behaviors.

Abstract

Despite their central role in fault detection, test oracles remain challenging to construct effectively. Recent learning based methods address this challenge by automatically generating test assertions, yet even if syntactically correct, they are often ineffective in revealing bugs. Rather than generating assertions, this study explores a different approach by training a model to directly predict whether a given test prefix passes or fails. We present FOCAL, an emerging code LLM-based discriminative oracle predictor. It learns from labeled pairs of test prefixes and methods under test, employs losses that emphasize failing cases during training, and grounds its predictions in statement level behavioral evidence. Compared with the baseline method SEER, we substantially improve performance on failing cases for unseen projects and provide richer explanations. A preliminary evaluation on fault-detection benchmarks and automated test-generation artifacts shows that our approach is highly accurate within its training distribution and substantially improves failure detection on previously unseen projects where prior discriminative oracles collapse. Moreover, the highlighted statements are supported by behavioral explanation checks. These early results suggest that fail-aware discriminative oracle prediction can complement existing approaches such as fuzzing, search-based testing, and LLM-based test generation. These techniques produce test prefixes at scale but often lack fault oriented oracles. In future work, FOCAL could take generated test prefixes and attach fault-aware predicted oracles to them, turning high-volume input generation into executable tests that are more likely to expose semantic failures.

Problem

• test oracle
• failure prediction
• discriminative model
• explainability
• unseen projects

Innovation

• fail-aware oracle
• discriminative prediction
• explainable AI
• test oracle
• large language model