Delineating Knowledge Boundaries for Honest Large Vision-Language Models

2026-04-29

AI Summary

This work addresses the tendency of large vision-language models to generate factually hallucinated responses and fail to abstain when confronted with questions beyond their knowledge scope. To mitigate this, the authors propose a knowledge boundary identification mechanism based on multi-sample consistency probing, integrated with supervised fine-tuning and preference optimization techniques such as DPO and ORPO to train models to reliably refuse unknown queries. A model-specific Visual-Idk dataset is constructed to support this approach. Experimental results demonstrate that the model does not merely memorize refusal patterns but genuinely learns to recognize its knowledge boundaries. The method improves the factual response rate from 57.9% to 67.3% and exhibits strong generalization and honesty on out-of-distribution domains, including medical and perceptual tasks.

Abstract

Large Vision-Language Models (VLMs) have achieved remarkable multimodal performance yet remain prone to factual hallucinations, particularly in long-tail or specialized domains. Moreover, current models exhibit a weak capacity to refuse queries that exceed their parametric knowledge. In this paper, we propose a systematic framework to enhance the refusal capability of VLMs when facing such unknown questions. We first curate a model-specific "Visual-Idk" (Visual-I don't know) dataset, leveraging multi-sample consistency probing to distinguish between known and unknown facts. We then align the model using supervised fine-tuning followed by preference-aware optimization (e.g., DPO, ORPO) to effectively delineate its knowledge boundaries. Results on the Visual-Idk dataset show our method improves the Truthful Rate from 57.9\% to 67.3\%. Additionally, internal probing also demonstrates that the model genuinely recognizes its boundaries instead of just memorizing refusal patterns. Our framework further generalizes to out-of-distribution medical and perceptual domains, providing a robust path toward more trustworthy and prudent visual assistants.

Problem

• Large Vision-Language Models
• factual hallucinations
• refusal capability
• knowledge boundaries
• unknown queries

Innovation

• Visual-Idk
• knowledge boundary delineation
• hallucination mitigation
• preference-aware optimization
• multimodal refusal capability