A Survey of Fuzzing Open-Source Operating Systems

2025-02-17

AI Summary

Open-source operating system (OS) vulnerability detection faces significant challenges, as existing fuzzing techniques lack systematic survey and evaluation across multi-layered OS components—including kernels, file systems, device drivers, and hypervisors. This paper presents the first comprehensive survey of OS fuzzing (OSF), establishing a multidimensional taxonomy grounded in the fuzzing lifecycle (seed generation, mutation, execution, and feedback). It uniformly analyzes common bottlenecks—such as state sensitivity, heterogeneous interface modeling, and cross-layer taint propagation—as well as scenario-specific challenges across four core OS layers. Our analysis identifies fundamental limitations in scalability, coverage-guidance precision, and system-call semantic modeling. Based on these insights, we propose five key future directions: protocol-aware fuzzing for heterogeneous interfaces, state-sensitive mutation strategies, cross-layer taint tracking, lightweight kernel coverage feedback, and synergistic optimization of symbolic execution and fuzzing.

Technology Category

• Search and Optimization: Distributed Search
• Computer Vision: Other Foundations of Computer Vision
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Vulnerabilities in open-source operating systems (OSs) pose substantial security risks to software systems, making their detection crucial. While fuzzing has been an effective vulnerability detection technique in various domains, OS fuzzing (OSF) faces unique challenges due to OS complexity and multi-layered interaction, and has not been comprehensively reviewed. Therefore, this work systematically surveys the state-of-the-art OSF techniques, categorizes them based on the general fuzzing process, and investigates challenges specific to kernel, file system, driver, and hypervisor fuzzing. Finally, future research directions for OSF are discussed.

Problem

• Detecting vulnerabilities in open-source OSs
• Challenges of fuzzing in complex OS environments
• Reviewing state-of-the-art OS fuzzing techniques

Innovation

• Systematic survey of OS fuzzing
• Categorizes fuzzing process techniques
• Investigates kernel, file system challenges