ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI

2026-06-16

AI Summary

Existing embodied question-answering benchmarks are susceptible to visual or linguistic shortcuts, hindering precise evaluation of agents’ reasoning capabilities in spatial relations, action planning, procedural understanding, and human intent. To address this, this work proposes ERQA-Plus—a structured diagnostic benchmark encompassing five dimensions: perception, action, social interaction, navigation, and commonsense reasoning—with 1,766 question-answer pairs derived from 711 robot-centric images. We introduce the first fine-grained taxonomy tailored for embodied reasoning and employ a hybrid pipeline combining taxonomy-guided question generation, automated scoring, iterative refinement, and human validation to ensure assessments target deep reasoning rather than superficial pattern matching. Experiments show that even the strongest current model, Qwen3-VL-32B, achieves only 83.4% accuracy and exhibits significant deficits in spatial, procedural, and intention-based reasoning, revealing critical limitations in complex embodied reasoning.

Abstract

Generalist embodied agents require more than object recognition: they must reason about spatial relations, actions, procedures, human intentions, environmental constraints, and commonsense consequences from situated visual observations. Yet existing visual and embodied question answering benchmarks often provide limited control over the reasoning dependencies being tested, making it difficult to distinguish grounded embodied reasoning from shortcut-driven visual or linguistic pattern matching. We present ERQA-Plus, a diagnostic benchmark for reasoning in embodied AI. ERQA-Plus contains 1,766 question-answer instances grounded in 711 robot-centric images and organized according to a structured taxonomy spanning perceptual, action-centric, social-interaction, navigation-environmental, and contextual commonsense reasoning. The dataset is constructed using a multi-stage generation and validation pipeline that combines taxonomy-guided question generation, automatic quality judging, iterative revision, and human assessment to improve visual grounding, answer validity, and reasoning quality. We benchmark representative general-purpose vision-language models and embodied models, including LLaVA-NeXT-8B, Prismatic-7B, MiniCPM-V-4.5-8B, Qwen3-VL, RoboRefer-8B, and RoboBrain2.5-8B. Although the strongest model, Qwen3-VL-32B, achieves 83.4% overall accuracy and 61.4 SBERT score, category-level results reveal persistent weaknesses in spatial reasoning, procedural reasoning, event prediction, and intention inference. ERQA-Plus therefore provides a fine-grained evaluation framework for measuring not only whether embodied agents answer correctly, but also which forms of embodied reasoning they can and cannot perform reliably. The dataset is available https://huggingface.co/datasets/huggingdas/erqa-plus and the project page at https://github.com/LUNAProject22/erqa-plus.

Problem

• embodied reasoning
• visual question answering
• spatial reasoning
• commonsense reasoning
• diagnostic benchmark

Innovation

• embodied reasoning
• diagnostic benchmark
• structured taxonomy
• visual grounding
• reasoning evaluation