🤖 AI Summary
This work proposes the first agent-driven automated testing and repair framework targeting four classes of potential flaws—verified code, checker code, unverified code, and specifications—in verified compilers generated by coding agents. By leveraging the internal structure of compilers, the approach designs structure-aware, customized defect detection strategies and employs coding agents to automatically repair identified issues, with correctness ensured through formal verification and benchmark testing. Experimental evaluation on the Axon compiler demonstrates that the system effectively identifies and fixes defects without exhibiting reward-hacking behavior, thereby validating both the efficacy and safety of the proposed methodology.
📝 Abstract
We present an agent based automated testing and repair system for verified compilers that contain four kinds of code: verified code, checked code, unverified code, and specification. We present specialized defect detection techniques that exploit the structure present in such compilers. For each surfaced defect the system invokes a coding agent to repair the defect and validate the repair. We evaluate the system on the Axon compiler, a compiler completely generated by a coding agent operating under developer supervision. The compiler was validated during development on a small benchmark set (the Livermore benchmarks), raising the possibility that its coding agent reward hacked the compiler. We also evaluate the possibility that the repair system reward hacked the repairs and find no evidence of reward hacking in either the Axon compiler or the repairs. We present results that characterize the testing and repair effectiveness and discuss repair characteristics.