Reliability-Prioritized Fine-Grained Generation in Multimodal Large

2026-06-28

AI Summary

This work addresses the reliability challenges faced by multimodal large language models when generating fine-grained visual descriptions, where excessive specificity often compromises factual accuracy. The study systematically uncovers an inherent trade-off between description granularity and reliability, and introduces a reliability-prioritized preference optimization framework. This approach leverages GranFact—a novel multi-object image benchmark validated by human experts—alongside a hierarchical fine-grained evaluation algorithm and a training strategy based on Direct Preference Optimization. The method effectively enhances descriptive concreteness while preserving visual fidelity. Experimental results demonstrate that the proposed model significantly outperforms existing approaches on GranFact, achieving a balanced improvement in both generation quality and reliability.

Abstract

Multimodal large language models (MLLMs) are increasingly expected to generate fine-grained descriptions of visual content. However, we observe and theoretically show that generating fine-grained responses poses a reliability challenge, \textit{i.e.}, fine-grained generation is more error-prone than coarse-grained generation. This phenomenon suggests that models should generate the finest description that remains reliable rather than simply produce more specific outputs. To investigate this problem, we develop \textsc{GranFact}, a granularity-aware benchmark consisting of expert-verified multi-object images with coarse-to-fine category annotations. Then, we design a hierarchy-aware evaluation algorithm, which assesses both whether model predictions are visually correct and how specific the correct predictions are. We also propose a reliability-prioritized preference optimization method based on Direct Preference Optimization, which penalizes unreliable fine-grained claims while rewarding reliable specificity. Experiments on \textsc{GranFact} show that our method improves fine-grained generation while preserving reliability. Code and data are available \href{https://github.com/WeiWu2025/GranFact}{here}.

Problem

• fine-grained generation
• reliability
• multimodal large language models
• granularity
• visual description

Innovation

• fine-grained generation
• reliability-prioritized optimization
• multimodal large language models
• granularity-aware benchmark
• hierarchy-aware evaluation