VISOR: A Vision-Language Model-based Test Oracle for Testing Robot

2026-05-11

AI Summary

This work addresses the lack of general, automated test oracles in robotics by proposing VISOR—the first vision-language model (VLM)-based automated evaluation framework. VISOR assesses both task correctness and execution quality by analyzing the alignment between task execution videos and natural language instructions, overcoming the binary-output limitation of traditional symbolic oracles. It further incorporates an uncertainty quantification mechanism to reflect prediction confidence. Experiments across four tasks and over 1,000 videos reveal that VLMs such as GPT and Gemini exhibit complementary strengths—GPT achieves higher precision, while Gemini demonstrates superior recall. However, their predicted uncertainties show weak correlation with actual judgment accuracy, limiting their direct utility for reliability prediction.

Abstract

Testing robots requires assessing whether they perform their intended tasks correctly, dependably, and with high quality, a challenge known as the test oracle problem in software testing. Traditionally, this assessment relies on task-specific symbolic oracles for task correctness and on human manual evaluation of robot behavior, which is time-consuming, subjective, and error-prone. To address this, we propose VISOR, a Vision-Language Model (VLM)-based approach for automated test oracle assessment that eliminates the need of expensive human evaluations. VISOR performs automated evaluation of task correctness and quality, addressing the limitations of existing symbolic test oracles, which are task-specific and provide pass/fail judgments without explicitly quantifying task quality. Given the inherent uncertainty in VLMs, VISOR also explicitly quantifies its own uncertainty during test assessments. We evaluated VISOR using two VLMs, i.e., GPT and Gemini, across four robotic tasks on over 1,000 videos. Results show that Gemini achieves higher recall while GPT achieves higher precision. However, both models show low correlation between uncertainty and correctness, which prevents using uncertainty as a correctness predictor.

Problem

• test oracle
• robot testing
• vision-language model
• automated evaluation
• task correctness

Innovation

• Vision-Language Model
• Test Oracle
• Robot Testing
• Uncertainty Quantification
• Automated Evaluation