Tratto: A Neuro-Symbolic Approach to Deriving Axiomatic Test Oracles

2025-04-05

AI Summary

To address the lack of reliable oracles—specifically, axiomatized test oracles—in automated testing, this paper proposes a source-code- and documentation-aware neuro-symbolic approach. The method synergistically integrates syntax-constrained symbolic search with a multi-task fine-tuned Transformer model that jointly performs assertion discrimination and token-level assertion generation, augmented by a context-aware symbolic space pruning mechanism. It is the first to formally combine grammar-guided symbolic reasoning with deep semantic modeling, ensuring logical correctness while improving generation accuracy. Experimental results demonstrate an assertion generation accuracy of 73%, an F1 score of 61%, a threefold speedup over pure symbolic methods, and a false positive rate only one-tenth that of GPT-4 augmented with chain-of-thought prompting. These advances significantly enhance the reliability and practicality of axiomatized test oracles.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

This paper presents Tratto, a neuro-symbolic approach that generates assertions (boolean expressions) that can serve as axiomatic oracles, from source code and documentation. The symbolic module of Tratto takes advantage of the grammar of the programming language, the unit under test, and the context of the unit (its class and available APIs) to restrict the search space of the tokens that can be successfully used to generate valid oracles. The neural module of Tratto uses transformers fine-tuned for both deciding whether to output an oracle or not and selecting the next lexical token to incrementally build the oracle from the set of tokens returned by the symbolic module. Our experiments show that Tratto outperforms the state-of-the-art axiomatic oracle generation approaches, with 73% accuracy, 72% precision, and 61% F1-score, largely higher than the best results of the symbolic and neural approaches considered in our study (61%, 62%, and 37%, respectively). Tratto can generate three times more axiomatic oracles than current symbolic approaches, while generating 10 times less false positives than GPT4 complemented with few-shot learning and Chain-of-Thought prompting.

Problem

• Generates axiomatic test oracles from code and documentation
• Combines symbolic and neural modules for oracle generation
• Outperforms state-of-the-art in accuracy and precision

Innovation

• Neuro-symbolic approach for oracle generation
• Symbolic module restricts token search space
• Neural module uses transformers for token selection