ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models

2026-06-18

AI Summary

This work addresses the perceptual-action gap in multimodal large language models, which struggle to generate context-consistent actions under identical visual inputs. To this end, the authors introduce the ROSE benchmark—a novel evaluation framework that isolates perceptual-action consistency by holding visual inputs constant while varying region constraints and symbolic output requirements, thereby assessing fine-grained visual reasoning and action generation against implicit majority references. Through controlled tasks such as counting-coordinate coupling and global-click versus local-matching, experiments across nine state-of-the-art models reveal a performance drop of up to 44.5 percentage points in region-conditioned action tasks compared to counting, despite human accuracy reaching 98.8%. These findings expose a significant context-induced performance discontinuity and a task-specific bottleneck independent of localization capability.

Abstract

Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts. How reliably can a model turn the same visual evidence into the action required by the current context? To answer this question, we introduce \textsc{ROSE} (\textbf{R}eference-conditioned \textbf{O}ddity and \textbf{S}ymbolic \textbf{E}xecution), a controlled benchmark that holds the visual scene fixed while varying region constraints and required symbolic outputs. Through coupled counting and coordinate-action tasks, \textsc{ROSE} tests whether models can infer an implicit majority reference and act on the resulting fine-grained visual evidence under changing contexts. Across nine recent MLLMs, performance drops by as much as 44.5 percentage points from counting-oriented tasks to region-conditioned action, despite 98.8\% human performance. The gap persists on paired scenes and regions for which the same model returns the correct count, while global-click and matched local controls show that coordinate grounding explains only part of the loss, revealing a distinct, model-dependent bottleneck in turning shared visual evidence into context-specific actions.

Problem

• perception-to-action gap
• multimodal models
• visual reasoning
• context-dependent action
• symbolic execution

Innovation

• multimodal large language models
• perception-to-action gap
• context-conditioned reasoning
• symbolic execution
• visual grounding