Convincing Rationales for Visual Question Answering Reasoning

2024-02-06

arXiv.org

AI Summary

Existing VQA models are largely “black-box” systems that prioritize answer accuracy while neglecting the interpretability and reliability of reasoning processes. To address this, we propose CRVQA, the first end-to-end framework for jointly generating bimodal rationales—comprising both visual regions and textual fragments—to substantiate answer predictions. Methodologically, CRVQA employs a multimodal fusion architecture integrating visual attention-based region localization, token-level textual importance modeling, rationale–answer co-training, and a novel rationale consistency supervision mechanism. Evaluated across multiple benchmarks, CRVQA achieves a rationale support rate of 92.3% and yields an average 2.1% improvement in answer accuracy; it further demonstrates robust zero-shot transfer performance. We publicly release both the curated dataset and implementation code to advance research on interpretable VQA.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Visual Question Answering (VQA) is a challenging task of predicting the answer to a question about the content of an image. It requires deep understanding of both the textual question and visual image. Prior works directly evaluate the answering models by simply calculating the accuracy of the predicted answers. However, the inner reasoning behind the prediction is disregarded in such a"black box"system, and we do not even know if one can trust the predictions. In some cases, the models still get the correct answers even when they focus on irrelevant visual regions or textual tokens, which makes the models unreliable and illogical. To generate both visual and textual rationales next to the predicted answer to the given image/question pair, we propose Convincing Rationales for VQA, CRVQA. Considering the extra annotations brought by the new outputs, {CRVQA} is trained and evaluated by samples converted from some existing VQA datasets and their visual labels. The extensive experiments demonstrate that the visual and textual rationales support the prediction of the answers, and further improve the accuracy. Furthermore, {CRVQA} achieves competitive performance on generic VQA datatsets in the zero-shot evaluation setting. The dataset and source code will be released under https://github.com/lik1996/CRVQA2024.

Problem

• VQA lacks transparent reasoning behind answer predictions
• Models may focus on irrelevant regions, causing unreliability
• Proposing MRVQA to generate visual and textual rationales

Innovation

• Generates visual and textual rationales for VQA
• Uses converted samples from existing VQA datasets
• Improves accuracy with multimodal rationale support