Search-Based Metamorphic Testing of Vision-Language Models in Autonomous Underwater Robotic Software

2026-09-15

AI Summary

为评估和提高视觉-语言模型在自主水下机器人软件中的适用性和可靠性，提出一种基于搜索的变形测试方法MetaVLM，通过最小化图像变换揭示模型错误。

Abstract

Our industry partner focuses on quality assurance for industrial systems across multiple domains, including maritime systems, such as overwater vessels and autonomous underwater robots (AURs). Despite the strong performance of vision-language models (VLMs) in scene understanding, image captioning, and object recognition, their use in AUR software operating in underwater environments is underexplored. Therefore, in this context, it is important to evaluate the quality of VLMs for integration into AUR software and, so, automated software testing tools are needed to assess their suitability and improve their dependability. To this end, we propose a search-based metamorphic testing approach (MetaVLM) that identifies a minimal set of transformations on underwater images to induce incorrect model predictions, thereby revealing VLM failures. We employ NSGA-II as a multi-objective search algorithm and evaluate it over open-source VLMs, BLIP and CLIP, against a random search baseline. Results demonstrate the strengths and limitations of each VLM in the context of AUR software systems. Based on the results, we derive lessons for software engineering practitioners and researchers working on quality assurance of VLM-based software systems.

Problem

• Vision-Language Models
• Autonomous Underwater Robots
• Quality Assurance

Innovation

• search-based metamorphic testing
• vision-language models
• autonomous underwater robots
• NSGA-II
• software quality assurance