Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components

2025-10-24

AI Summary

Existing deep learning (DL) framework testing methods inadequately expose memory-related crashes in automotive deployment, primarily due to insufficient support for multi-input/multi-output tensors, multimodal data fusion, and hierarchical feature extraction. To address this, we propose a component-based testing methodology that assembles models from modular components—head, neck, and backbone—enabling fine-grained test case generation. Our approach introduces component-level selection, mutation, and genetic dynamic assembly to construct diverse, scenario-adapted models for autonomous driving. Integrated with a component repository, differential testing, and multimodal tensor processing, it enables efficient detection of memory-related defects in embedded DL frameworks. Experimental evaluation on mainstream automotive DL frameworks demonstrates a substantial improvement in memory defect detection rates and successfully uncovers multiple previously unknown crash vulnerabilities under real-world driving conditions.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Multi-modal Vision
• Application Domains: Mobility, Driving & Flight

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Deep learning (DL) plays a key role in autonomous driving systems. DL models support perception modules, equipped with tasks such as object detection and sensor fusion. These DL models enable vehicles to process multi-sensor inputs to understand complex surroundings. Deploying DL models in autonomous driving systems faces stringent challenges, including real-time processing, limited computational resources, and strict power constraints. To address these challenges, automotive DL frameworks (e.g., PaddleInference) have emerged to optimize inference efficiency. However, these frameworks encounter unique quality issues due to their more complex deployment environments, such as crashes stemming from limited scheduled memory and incorrect memory allocation. Unfortunately, existing DL framework testing methods fail to detect these quality issues due to the failure in deploying generated test input models, as these models lack three essential capabilities: (1) multi-input/output tensor processing, (2) multi-modal data processing, and (3) multi-level data feature extraction. These capabilities necessitate specialized model components, which existing testing methods neglect during model generation. To bridge this gap, we propose Scalpel, an automotive DL frameworks testing method that generates test input models at the model component level. Scalpel generates models by assembling model components (heads, necks, backbones) to support capabilities required by autonomous driving systems. Specifically, Scalpel maintains and updates a repository of model components, generating test inputs by selecting, mutating, and assembling them. Successfully generated models are added back to enrich the repository. Newly generated models are then deployed within the autonomous driving system to test automotive DL frameworks via differential testing.

Problem

• Testing automotive DL frameworks' unique quality issues
• Generating test models with multi-modal data processing capabilities
• Assembling model components for autonomous driving system requirements

Innovation

• Assembles model components for testing
• Generates test inputs via component mutation
• Uses differential testing on automotive frameworks