Instruction-Following Evaluation of Large Vision-Language Models

2025-12-29

AI Summary

Large Vision-Language Models (LVLMs) commonly suffer from degraded instruction-following capability after standard visual instruction tuning, primarily due to the neglect of output format constraints during training, leading to misaligned task understanding. Method: This work presents the first quantitative characterization of this degradation and introduces *explicit output format annotation*: a multi-dimensional instruction dataset incorporating explicit format constraints, integrated with controllable fine-tuning and structured prompt analysis. Contribution/Results: Experiments demonstrate that our method improves LVLMs’ instruction-following accuracy by an average of 23.6%, substantially restoring and enhancing their controllable generation capability—especially for complex, format-sensitive tasks. The core innovation lies in treating output format as an explicit supervisory signal within the visual instruction tuning paradigm, offering a scalable and deployment-friendly pathway to improve LVLM reliability and controllability.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Following the initial flourishing of large language models (LLMs), there has been a surge in proposed large vision-language models (LVLMs) that integrate LLMs with vision capabilities. However, it has been observed that LVLMs, after tuning to visual instruction using commonly used training datasets, often fail to exhibit the instruction-following ability that was present in the LLM before integration, leading to results in which they do not follow task instructions as expected. This study quantitatively demonstrates that LVLMs' instruction-following ability declines after fine-tuning and analyzes its underlying causes. In particular, we constructed new training datasets highlighting whether the output format is specified. Then, we investigated how explicitly indicating the output format during fine-tuning affects LVLMs' instruction-following ability. Our quantitative evaluation confirmed that LVLMs' instruction-following ability declines after fine-tuning with commonly used datasets. Furthermore, we found that LVLMs trained with datasets, including instructions on output format, tend to follow instructions more accurately than models that do not. These findings suggest that including samples with instructions on output format during (visual) instruction tuning may help mitigate the decline in instruction-following abilities.

Problem

• Evaluates instruction-following decline in vision-language models after fine-tuning
• Analyzes causes of reduced instruction adherence in integrated multimodal models
• Investigates training data impact on output format specification and model performance

Innovation

• Constructed datasets highlighting output format specification
• Investigated explicit output format indication during fine-tuning
• Included output format instructions to mitigate ability decline