Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

2026-07-10

AI Summary

Current evaluations of vision-language models are largely confined to simplistic scenarios, failing to adequately assess their capacity to understand complex social behaviors. To address this gap, this work introduces CSB—a novel benchmark dataset specifically designed for evaluating comprehension of intricate social interactions—and develops an analytical framework that categorizes visual reasoning errors into five types: object detection, recognition, hallucination, scene understanding, and spatial dependency. A systematic evaluation of nine state-of-the-art multimodal large language models (MLLMs) from 2017 to 2025 reveals that the best-performing models now approach human-level accuracy in overall description quality, substantially reducing most error types, though a notable gap persists in modeling spatial dependencies. Among the error categories, detection, recognition, and hallucination errors exhibit the strongest impact on overall performance.

Abstract

Vision language models (VLMs) have made remarkable progress in visual reasoning during the last decade. Most evaluations have used simple scenes (MS-COCO) that do not showcase complex human interactions or behaviors, only a handful of non-curated human descriptions as a benchmark, and have not focused on understanding the model's error types. Here, we introduce the Complex Social Behavior (CSB) dataset, containing 100 images depicting complex social interactions/behaviors. We analyze the progression of scene descriptions over a decade (2017-2025) of VLMs (four pre-Multimodal Large Language Models, MLLMs, and five MLLMs). We evaluate the accuracy of the models and 20 human descriptions relative to a gold standard on the CSB dataset and on a sample from MS-COCO. We analyzed five visual-cognitive error types: object detection, recognition, hallucination, scene understanding, and spatial dependence. The CSB dataset showed a more pronounced improvement than MS-COCO in scene description accuracy, with pre-MLLMs achieving much lower accuracy than the bottom-ranked human descriptions and MLLMs attaining accuracies similar to the top-ranked human descriptions. We show that MLLMs have eliminated the gap in scene description accuracy between simpler MS-COCO scenes and scenes depicting complex behaviors (CSB). MLLMs have almost eliminated all error types in our tested datasets, except for occasionally relying on different image regions for scene descriptions than humans do (spatial dependence error). We also show that detection, recognition, and hallucination errors have the highest impact on scene description accuracy. Together, our findings provide a more thorough evaluation of how visual language models have advanced over the last decade.

Problem

• vision-language models
• complex social behavior
• visual-cognitive errors
• scene description accuracy
• model evaluation

Innovation

• Complex Social Behavior dataset
• vision-language models
• visual-cognitive errors
• multimodal large language models
• scene description accuracy