Iterative Repair with Weak Verifiers for Few-shot Transfer in KBQA with Unanswerability

2024-06-20

AI Summary

This work addresses the challenge of identifying and handling unanswerable questions in Knowledge Base Question Answering (KBQA), particularly under realistic low-resource domain settings with scarce annotated data. To relax the prevalent “all-questions-are-answerable” assumption, the authors propose the first few-shot transfer learning task specifically for unanswerable questions. Their method introduces an iterative repair mechanism integrating strong and weak verifiers, coupled with a self-consistency-based discriminative framework tailored to unanswerability. Built upon FuSIC-KBQA, the approach incorporates multi-verifier feedback, self-consistency evaluation, and few-shot transfer techniques. The paper contributes two novel benchmark datasets for unanswerability assessment. Empirically, it achieves significant improvements over mainstream large language models and supervised state-of-the-art methods on the unanswerable few-shot transfer task, while also establishing new state-of-the-art performance on the answerable few-shot transfer task.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web question answering
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Community question answering

Abstract

Real-world applications of KBQA require models to handle unanswerable questions with a limited volume of in-domain labeled training data. We propose the novel task of few-shot transfer for KBQA with unanswerable questions and contribute two new datasets for performance evaluation. We present FUn-FuSIC - a novel solution for our task that extends FuSIC KBQA, the state-of-the-art few-shot transfer model for answerable-only KBQA. We first note that FuSIC-KBQA's iterative repair makes a strong assumption that all questions are unanswerable. As a remedy, we propose Feedback for Unanswerability (FUn), which uses iterative repair using feedback from a suite of strong and weak verifiers, and an adaptation of self consistency for unanswerabilty to better assess the answerability of a question. Our experiments show that FUn-FuSIC significantly outperforms suitable adaptations of multiple LLM based and supervised SoTA models on our task, while establishing a new SoTA for answerable few-shot transfer as well.

Problem

• Handles unanswerable KBQA questions
• Few-shot transfer with limited data
• Proposes FUn-FuSIC iterative repair solution

Innovation

• Iterative repair with feedback
• Weak verifiers for unanswerability
• Few-shot transfer KBQA enhancement