Reflect then Learn: Active Prompting for Information Extraction Guided by Introspective Confusion

2025-08-09

AI Summary

In few-shot information extraction, large language models (LLMs) exhibit strong dependence on the quality of in-context examples; however, conventional example selection strategies neglect LLMs’ dual confusion—between structured format generation and semantic understanding. To address this, we propose APIE, an Active Prompting framework for Information Extraction, which introduces *introspective confusion*—a novel mechanism jointly modeling format uncertainty and content uncertainty—to enable LLMs to autonomously identify the most challenging examples for prompt construction. APIE integrates dual-dimensional uncertainty quantification, active prompting learning, and few-shot in-context learning. Evaluated on four standard benchmarks, APIE consistently outperforms strong baselines, achieving average F1-score gains of 3.2–5.8 percentage points, alongside improved accuracy and robustness.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Active Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) show remarkable potential for few-shot information extraction (IE), yet their performance is highly sensitive to the choice of in-context examples. Conventional selection strategies often fail to provide informative guidance, as they overlook a key source of model fallibility: confusion stemming not just from semantic content, but also from the generation of well-structured formats required by IE tasks. To address this, we introduce Active Prompting for Information Extraction (APIE), a novel active prompting framework guided by a principle we term introspective confusion. Our method empowers an LLM to assess its own confusion through a dual-component uncertainty metric that uniquely quantifies both Format Uncertainty (difficulty in generating correct syntax) and Content Uncertainty (inconsistency in extracted semantics). By ranking unlabeled data with this comprehensive score, our framework actively selects the most challenging and informative samples to serve as few-shot exemplars. Extensive experiments on four benchmarks show that our approach consistently outperforms strong baselines, yielding significant improvements in both extraction accuracy and robustness. Our work highlights the critical importance of a fine-grained, dual-level view of model uncertainty when it comes to building effective and reliable structured generation systems.

Problem

• Addresses sensitivity to in-context examples in few-shot IE
• Quantifies both format and content uncertainty in LLMs
• Selects challenging samples to improve extraction accuracy

Innovation

• Active Prompting framework with introspective confusion
• Dual-component uncertainty metric for format and content
• Ranking unlabeled data to select challenging exemplars