Zero-Shot Belief: A Hard Problem for LLMs

2025-02-12

AI Summary

This work addresses zero-shot source-target belief prediction—jointly identifying events, localizing sources, and classifying belief polarity—on the FactBank dataset. We systematically evaluate mainstream open-source, closed-source, and reasoning-augmented LLMs, revealing their fundamental limitations on this fine-grained structured prediction task for the first time. To overcome these limitations, we propose two novel paradigms: (1) unified end-to-end LLM-based generative prediction, and (2) a hybrid framework integrating a DeBERTa-based sequence tagger with an LLM for collaborative inference. Our approach achieves new state-of-the-art performance on FactBank, enables comprehensive fine-grained error analysis, and successfully transfers to the Italian ModaFact dataset, demonstrating cross-lingual generalization. The core contributions are: (i) establishing the first zero-shot benchmark for belief structure parsing; (ii) rigorously characterizing methodological boundaries of current LLMs on structured belief extraction; and (iii) providing an extensible, multi-stage modeling paradigm for belief analysis.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present two LLM-based approaches to zero-shot source-and-target belief prediction on FactBank: a unified system that identifies events, sources, and belief labels in a single pass, and a hybrid approach that uses a fine-tuned DeBERTa tagger for event detection. We show that multiple open-sourced, closed-source, and reasoning-based LLMs struggle with the task. Using the hybrid approach, we achieve new state-of-the-art results on FactBank and offer a detailed error analysis. Our approach is then tested on the Italian belief corpus ModaFact.

Problem

• Zero-shot belief prediction
• LLM-based approaches
• FactBank and ModaFact

Innovation

• Unified system for event identification
• Hybrid DeBERTa tagger approach
• State-of-the-art results on FactBank