SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles

2026-01-21

AI Summary

This work addresses the high cost, susceptibility to false positives, and limited adaptability of manually crafted oracles in JavaScript differential fuzzing. To overcome these challenges, the study introduces a novel multi-agent architecture powered by large language models for automated oracle construction. The proposed system employs specialized sub-agents that collaboratively analyze execution logs, retrieve relevant specification evidence, and synthesize decisions to effectively filter noise. Evaluated on historical benchmarks, the approach achieves a recall of 0.84 with a 18% false positive rate. Compared to a Gemini 2.5 Pro baseline, it delivers a fourfold speedup in analysis and reduces API costs by an order of magnitude. Furthermore, the method has uncovered previously unknown specification-level bugs in major JavaScript engines, including V8, JavaScriptCore, and GraalJS.

Technology Category

• Search and Optimization: Distributed Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Differential fuzzers detect bugs by executing identical inputs across distinct implementations of the same specification, such as JavaScript interpreters. Validating the outputs requires an oracle and for differential testing of JavaScript, these are constructed manually, making them expensive, time-consuming, and prone to false positives. Worse, when the specification evolves, this manual effort must be repeated. Inspired by the success of agentic systems in other SE domains, this paper introduces SmartOracle. SmartOracle decomposes the manual triage workflow into specialized Large Language Model (LLM) sub-agents. These agents synthesize independently gathered evidence from terminal runs and targeted specification queries to reach a final verdict. For historical benchmarks, SmartOracle achieves 0.84 recall with an 18% false positive rate. Compared to a sequential Gemini 2.5 Pro baseline, it improves triage accuracy while reducing analysis time by 4$\times$ and API costs by 10$\times$. In active fuzzing campaigns, SmartOracle successfully identified and reported previously unknown specification-level issues across major engines, including bugs in V8, JavaScriptCore, and GraalJS. The success of SmartOracle's agentic architecture on Javascript suggests it might be useful other software systems- a research direction we will explore in future work.

Problem

• differential oracle
• fuzzing
• JavaScript
• false positives
• specification evolution

Innovation

• agentic system
• differential fuzzing
• LLM-based oracle
• specification validation
• automated triage