CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering

2025-01-02

AI Summary

Existing vision-language models (VLMs) often generate erroneous answers to unanswerable visual question-answering (VQA) questions—those referencing objects absent from the image—undermining their reliability. To address this, we propose a lightweight, plug-and-play unanswerability detection framework that **introduces CLIP for the first time as a discriminator in VQA rejection**. Our method freezes the original VLM backbone and fine-tunes only a small number of adapter layers, integrating cross-modal alignment modeling with confidence calibration derived from multiple-choice VQA outputs. Crucially, it requires no modification to the pretrained weights. Evaluated on the MM-UPD benchmark, our approach achieves state-of-the-art unanswerability detection performance while **fully preserving LLaVA’s original VQA accuracy and multi-task generalization capability**, thus balancing robustness and practical deployability.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent Vision-Language Models (VLMs) have demonstrated remarkable capabilities in visual understanding and reasoning, and in particular on multiple-choice Visual Question Answering (VQA). Still, these models can make distinctly unnatural errors, for example, providing (wrong) answers to unanswerable VQA questions, such as questions asking about objects that do not appear in the image. To address this issue, we propose CLIP-UP: CLIP-based Unanswerable Problem detection, a novel lightweight method for equipping VLMs with the ability to withhold answers to unanswerable questions. By leveraging CLIP to extract question-image alignment information, CLIP-UP requires only efficient training of a few additional layers, while keeping the original VLMs' weights unchanged. Tested across LLaVA models, CLIP-UP achieves state-of-the-art results on the MM-UPD benchmark for assessing unanswerability in multiple-choice VQA, while preserving the original performance on other tasks.

Problem

• Visual-Language Models
• Image Question Answering
• Accuracy Improvement

Innovation

• CLIP-UP
• CLIP technology
• Visual and Language Model Enhancement