Psyzkaller: Learning from Historical and On-the-Fly Execution Data for Smarter Seed Generation in OS kernel Fuzzing

2025-10-09

AI Summary

Existing kernel fuzzers (e.g., Syzkaller) struggle to generate valid system call sequences satisfying system call dependency relationships (SDRs), resulting in high seed validation failure rates and insufficient path exploration depth. To address this, we propose an implicit SDR learning method based on an N-gram model, which jointly leverages the Dongting historical dataset and real-time execution traces to construct a dynamic Choice Table; we further design a bidirectional random walk strategy to generate seeds with high validity and diversity. Evaluated on three Linux kernel versions, our approach achieves 4.6–7.0% higher code coverage and triggers 110.4–187.2% more crashes than Syzkaller. It discovers eight previously unknown vulnerabilities—Syzkaller identifies only one—demonstrating significantly enhanced deep-path exploration capability and vulnerability detection efficiency.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Fuzzing has become a cornerstone technique for uncovering vulnerabilities and enhancing the security of OS kernels. However, state-of-the-art kernel fuzzers, including the de facto standard Syzkaller, struggle to generate valid syscall sequences that respect implicit Syscall Dependency Relations (SDRs). Consequently, many generated seeds either fail kernel validation or cannot penetrate deep execution paths, resulting in significant inefficiency. We hypothesize that SDRs can be effectively learned from both historic and present kernel execution data, and that incorporating these learned relations into fuzzing can substantially improve seed validity and diversity. To validate this, we propose an approach that utilizes the N-gram model to mine SDRs from the Dongting dataset-one of the largest Linux kernel execution datasets available-as well as from execution traces collected on the fly during fuzzing. The resulting model is used to continuously augment the Choice Table of Syzkaller to improve its seed generation and demonstrably increases the Shannon Entropy of the Choice Table throughout fuzzing, reflecting more empirically-grounded choices in expanding syscall sequences into valid and diverse seeds. In addition, we introduce a Random Walk strategy that instructs Syzkaller to construct seeds in a bidirectional manner to further diversify the generated seeds. We implement our approach in a prototype, Psyzkaller, built on top of Syzkaller. Experiments on three representative Linux kernel versions show that Psyzkaller improves Syzkaller's code coverage by 4.6%-7.0% in 48-hour fuzzing, while triggering 110.4%-187.2% more crashes. Moreover, our investigation shows that Psyzkaller discovered eight previously unknown kernel vulnerabilities, compared to only one found by Syzkaller.

Problem

• Learning syscall dependencies from historical and real-time kernel execution data
• Improving seed validity and diversity in OS kernel fuzzing
• Enhancing code coverage and vulnerability discovery in kernel fuzzing

Innovation

• Mines syscall dependencies from historical and real-time execution data
• Uses N-gram model to enhance seed generation in kernel fuzzing
• Implements bidirectional random walk strategy for seed diversification