GAPMAP: Mapping Scientific Knowledge Gaps in Biomedical Literature Using Large Language Models

2025-10-28

AI Summary

Existing approaches struggle to identify implicit research gaps—unstated knowledge deficiencies—in biomedical literature. Method: This study introduces “implicit knowledge gap inference” as a novel task and proposes TABI, a Toulmin-Abductive reasoning–based framework that enables structured, verifiable inference. TABI integrates paragraph- and full-text inputs, domain-adapted prompting, and human-in-the-loop verification, and is compatible with major closed- and open-weight LLMs (e.g., GPT, Llama, Gemma 2). Contribution/Results: Evaluated on nearly 1,500 biomedical papers, the approach demonstrates strong generalization across models; performance scales positively with model size. Results validate TABI’s practical utility for research topic selection, science policy formulation, and funding decision support, establishing its deployability in real-world scholarly workflows.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Scientific progress is driven by the deliberate articulation of what remains unknown. This study investigates the ability of large language models (LLMs) to identify research knowledge gaps in the biomedical literature. We define two categories of knowledge gaps: explicit gaps, clear declarations of missing knowledge; and implicit gaps, context-inferred missing knowledge. While prior work has focused mainly on explicit gap detection, we extend this line of research by addressing the novel task of inferring implicit gaps. We conducted two experiments on almost 1500 documents across four datasets, including a manually annotated corpus of biomedical articles. We benchmarked both closed-weight models (from OpenAI) and open-weight models (Llama and Gemma 2) under paragraph-level and full-paper settings. To address the reasoning of implicit gaps inference, we introduce extbf{small TABI}, a Toulmin-Abductive Bucketed Inference scheme that structures reasoning and buckets inferred conclusion candidates for validation. Our results highlight the robust capability of LLMs in identifying both explicit and implicit knowledge gaps. This is true for both open- and closed-weight models, with larger variants often performing better. This suggests a strong ability of LLMs for systematically identifying candidate knowledge gaps, which can support early-stage research formulation, policymakers, and funding decisions. We also report observed failure modes and outline directions for robust deployment, including domain adaptation, human-in-the-loop verification, and benchmarking across open- and closed-weight models.

Problem

• Identifying explicit and implicit knowledge gaps in biomedical literature
• Developing structured reasoning methods for implicit gap inference
• Benchmarking LLM performance across different model architectures

Innovation

• Detects implicit knowledge gaps using LLMs
• Introduces TABI scheme for structured reasoning
• Benchmarks open and closed weight model performance