Mut4All: Fuzzing Compilers via LLM-Synthesized Mutators Learned from Bug Reports

2025-07-25

AI Summary

Manually designing mutation operators for modern compilers (e.g., Rust, C++) is costly and poorly generalizable due to complex language features such as templates and macros. Method: This paper introduces the first language-agnostic, LLM-driven fuzzing framework. It automatically learns mutation patterns from 1,000 real-world bug reports and synthesizes mutation operators via a three-agent collaborative architecture—*Invent*, *Implement*, and *Refine*—guided by unit-test feedback and fine-grained code modification modeling using GPT-4o. Contribution/Results: At an average cost of $0.08 per operator, the framework synthesizes 722 effective mutation operators. These detect 62 vulnerabilities in Rust compilers and 34 in C++ compilers, including 38 previously unknown Rust bugs. This work pioneers deep integration of bug-report knowledge with large language models, significantly improving cross-language vulnerability detection efficiency and coverage.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Mutation-based fuzzing is effective for uncovering compiler bugs, but designing high-quality mutators for modern languages with complex constructs (e.g., templates, macros) remains challenging. Existing methods rely heavily on manual design or human-in-the-loop correction, limiting scalability and cross-language generalizability. We present Mut4All, a fully automated, language-agnostic framework that synthesizes mutators using Large Language Models (LLMs) and compiler-specific knowledge from bug reports. It consists of three agents: (1) a mutator invention agent that identifies mutation targets and generates mutator metadata using compiler-related insights; (2) a mutator implementation synthesis agent, fine-tuned to produce initial implementations; and (3) a mutator refinement agent that verifies and corrects the mutators via unit-test feedback. Mut4All processes 1000 bug reports (500 Rust, 500 C++), yielding 319 Rust and 403 C++ mutators at ~$0.08 each via GPT-4o. Our customized fuzzer, using these mutators, finds 62 bugs in Rust compilers (38 new, 7 fixed) and 34 bugs in C++ compilers (16 new, 1 fixed). Mut4All outperforms existing methods in both unique crash detection and coverage, ranking first on Rust and second on C++.

Problem

• Automating mutator design for compiler fuzzing with LLMs
• Overcoming manual mutator limitations in complex languages
• Enhancing bug detection in Rust and C++ compilers

Innovation

• LLM-synthesized mutators from bug reports
• Automated mutator refinement via unit-test feedback
• Language-agnostic framework for compiler fuzzing