Concurrency Testing in the Linux Kernel via eBPF

2025-04-30

AI Summary

Linux kernel concurrency bugs are notoriously elusive and difficult to reproduce; existing testing approaches suffer from high performance overhead, poor maintainability, and limited scalability. This paper proposes LACE, a lightweight controlled-concurrency testing framework. LACE introduces the first eBPF-based kernel-level scheduling control mechanism, enabling fine-grained thread scheduling and safe injection of scheduling points without hypervisor modification. It further designs a two-phase mutation strategy that synergistically combines scheduling perturbation with concurrent fuzzing. Compared to state-of-the-art kernel concurrency fuzzers, LACE achieves a 38% increase in branch coverage, reduces runtime overhead by 57%, and accelerates bug exposure by 11.4×. Empirical evaluation uncovered eight previously unknown concurrency bugs, six of which have been confirmed by Linux kernel developers.

Technology Category

• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Machine Learning: Kernel Methods
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Concurrency is vital for our critical software to meet modern performance requirements, yet concurrency bugs are notoriously difficult to detect and reproduce. Controlled Concurrency Testing (CCT) can make bugs easier to expose by enabling control over thread interleavings and systematically exploring the interleaving space through scheduling algorithms. However, existing CCT solutions for kernel code are heavyweight, leading to significant performance, maintainability and extensibility issues. In this work, we introduce LACE, a lightweight CCT framework for kernel code empowered by eBPF. Without hypervisor modification, LACE features a custom scheduler tailored for CCT algorithms to serialize non-determistic thread execution into a controlled ordering. LACE also provides a mechanism to safely inject scheduling points into the kernel for fine-grained control. Furthermore, LACE employs a two-phase mutation strategy to integrate the scheduler with a concurrency fuzzer, allowing for automated exploration of both the input and schedule space. In our evaluation, LACE achieves 38% more branches, 57% overhead reduction and 11.4$ imes$ speed-up in bug exposure compared to the state-of-the-art kernel concurrency fuzzers. Our qualitative analysis also demonstrates the extensibility and maintainability of LACE. Furthermore, LACE discovers eight previously unknown bugs in the Linux kernel, with six confirmed by developers.

Problem

• Detecting concurrency bugs in Linux kernel efficiently
• Reducing performance overhead in kernel concurrency testing
• Improving maintainability and extensibility of concurrency testing tools

Innovation

• Lightweight CCT framework using eBPF
• Custom scheduler for controlled thread ordering
• Two-phase mutation for input and schedule exploration