Search-based Testing of Vision Language Models for In-Car Scene Understanding

2026-07-02

AI Summary

This work addresses the challenges of scarce real-world data and inadequate testing methodologies in the early development of in-vehicle vision-language models (VLMs), which often lead to erroneous or incomplete outputs. To tackle this, the authors propose ISU-Test, a novel approach that integrates search-based testing with rendering-based scene generation for the first time. By leveraging optimization algorithms such as genetic algorithms, ISU-Test automatically tunes in-cabin scene parameters to efficiently produce diverse test cases, enabling systematic evaluation of VLM reliability in visual question answering and image captioning tasks. Experimental results demonstrate that ISU-Test achieves up to a 10× higher fault detection rate and 3.6× greater coverage compared to random generation, significantly outperforming existing baselines and establishing a scalable, automated framework for validating in-vehicle VLMs.

Abstract

In the automotive domain, in-car scene understanding (ISU) enables the detection of safety-critical events, such as driver distraction, and supports drivers or passengers by analyzing the in-car scene and adapting the environment (e.g., ambient lighting). The industry is increasingly exploring vision-language models (VLMs) to interpret camera-recorded in-car scenes and extract information for downstream reasoning tasks. However, VLMs may generate incomplete, erroneous, or misleading scene descriptions, highlighting the need for systematic testing. Collecting real in-vehicle data is costly, difficult to scale, and often infeasible, particularly in early design stages. In this paper, we present ISU-Test, an automated testing approach that combines rendering-based scene generation with search-based testing to evaluate ISU systems. By framing testing as an optimization problem and systematically modifying scene parameters, our method generates diverse in-car scenarios and explores a wide range of configurations. We evaluate ISU-Test on both an industrial prototype and open-source VLMs across two case studies: question answering and captioning, comparing against randomized scenario generation. Results show that ISU-Test significantly outperforms the baseline, achieving up to 10 times higher failure rates and up to 3.6 times higher failure coverage.

Problem

• in-car scene understanding
• vision-language models
• systematic testing
• automated testing
• safety-critical events

Innovation

• search-based testing
• vision-language models
• in-car scene understanding
• rendering-based generation
• automated testing