Collage: Decomposable Rapid Prototyping for Information Extraction on Scientific PDFs

2024-10-30

arXiv.org

AI Summary

Scientific PDF information extraction tools suffer from inconsistent input formats, opaque “black-box” behavior, poor fault tolerance, and limited format support—hindering literature analysis efficiency for non-NLP researchers. To address these challenges, we propose the first modular information extraction experimental framework specifically designed for scientific PDFs, enabling model-level decoupling, fine-grained intermediate-state visualization, and unified cross-model evaluation. The framework integrates Hugging Face token classifiers, diverse large language models (LLMs), and domain-specific models within a PDF processing pipeline comprising layout-aware parsing, text reconstruction, and semantic alignment. Evaluated on materials science literature review tasks, it significantly reduces model trial-and-error overhead, improves error attribution accuracy, and enhances system interpretability. The platform delivers a debuggable, reusable, plug-and-play rapid prototyping capability for scientific information extraction, empowering domain experts without NLP expertise.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent years in NLP have seen the continued development of domain-specific information extraction tools for scientific documents, alongside the release of increasingly multimodal pretrained transformer models. While the opportunity for scientists outside of NLP to evaluate and apply such systems to their own domains has never been clearer, these models are difficult to compare: they accept different input formats, are often black-box and give little insight into processing failures, and rarely handle PDF documents, the most common format of scientific publication. In this work, we present Collage, a tool designed for rapid prototyping, visualization, and evaluation of different information extraction models on scientific PDFs. Collage allows the use and evaluation of any HuggingFace token classifier, several LLMs, and multiple other task-specific models out of the box, and provides extensible software interfaces to accelerate experimentation with new models. Further, we enable both developers and users of NLP-based tools to inspect, debug, and better understand modeling pipelines by providing granular views of intermediate states of processing. We demonstrate our system in the context of information extraction to assist with literature review in materials science.

Problem

• Difficulty comparing multimodal NLP models for scientific PDFs
• Lack of tools for prototyping and evaluating extraction models
• Challenges in debugging and understanding NLP processing pipelines

Innovation

• Rapid prototyping tool for PDF information extraction
• Supports HuggingFace and LLMs out of the box
• Provides granular views for debugging pipelines