Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis

2026-06-17

AI Summary

This study addresses the lack of systematic evaluation and attribution analysis of large vision-language models (LVLMs) on fine-grained image understanding tasks. To this end, the authors construct FG-BMK, a fine-grained benchmark comprising 1.01 million questions and 280,000 images, and introduce a novel dual-paradigm evaluation framework that integrates human-centric and machine-oriented perspectives. By combining fine-grained visual question answering, vision-language perturbation analysis, and cross-model comparisons, the framework jointly diagnoses bottlenecks in semantic recognition and visual discrimination. The findings reveal that LVLM failures stem from multiple intertwined factors—including inadequate visual representations, semantic grounding biases, modality alignment deficiencies, and missing fine-grained knowledge—thereby offering interpretable, evidence-based guidance for future data curation and model design.

Abstract

Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal perception and reasoning capabilities. While numerous benchmarks have evaluated LVLMs from holistic or task-specific perspectives, their capabilities on fine-grained image tasks-fundamental to computer vision-remain insufficiently understood. To address this gap, we introduce FG-BMK, a comprehensive fine-grained evaluation benchmark containing 1.01 million questions and 0.28 million images, covering diverse scenarios from common object-centric domains to specialized domains. FG-BMK jointly evaluates dialogue-level fine-grained semantic recognition and feature-level visual discriminability through human-oriented and machine-oriented paradigms, enabling diagnostic analysis of whether LVLM failures arise from insufficient visual representations, weak visual-to-semantic grounding, or limited fine-grained knowledge. Through extensive experiments on a diverse set of representative LVLMs/VLMs, we find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge. We further analyze training design factors for improving fine-grained capabilities and examine how visual and linguistic perturbations affect LVLM predictions. These findings provide diagnostic insights into the limitations of current LVLMs and offer guidance for future data construction and model design in developing more reliable LVLMs for fine-grained visual tasks. Our code is open-source and available at https://fg-bmk.github.io/.

Problem

• Large Vision-Language Models
• fine-grained image tasks
• benchmarking
• visual recognition
• multimodal evaluation

Innovation

• fine-grained evaluation
• vision-language models
• diagnostic benchmarking
• visual-semantic grounding
• multimodal representation