DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding

2024-08-27

AI Summary

Existing multimodal large language models (MLLMs) face critical bottlenecks—excessive computational overhead or insufficient vision–language fusion—when tackling text-dense, layout-complex document understanding. To address this, we propose DocLayLLM, a lightweight MLLM with three key innovations: (1) a layout-aware visual patch encoding scheme integrating 2D positional embeddings; (2) native reuse of the LLM’s tokenization and semantic encoding capabilities for OCR-derived text, eliminating the need for dedicated OCR modules; and (3) the first chain-of-thought (CoT) pretraining framework coupled with CoT annealing, deeply embedding structured reasoning into multimodal modeling. Evaluated on multiple text-intensive document understanding benchmarks, DocLayLLM achieves state-of-the-art performance using significantly lower training resources than both OCR-dependent and OCR-free approaches. The code and models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Text-rich document understanding (TDU) requires comprehensive analysis of documents containing substantial textual content and complex layouts. While Multimodal Large Language Models (MLLMs) have achieved fast progress in this domain, existing approaches either demand significant computational resources or struggle with effective multi-modal integration. In this paper, we introduce DocLayLLM, an efficient multi-modal extension of LLMs specifically designed for TDU. By lightly integrating visual patch tokens and 2D positional tokens into LLMs' input and encoding the document content using the LLMs themselves, we fully take advantage of the document comprehension capability of LLMs and enhance their perception of OCR information. We have also deeply considered the role of chain-of-thought (CoT) and innovatively proposed the techniques of CoT Pre-training and CoT Annealing. Our DocLayLLM can achieve remarkable performances with lightweight training settings, showcasing its efficiency and effectiveness. Experimental results demonstrate that our DocLayLLM outperforms existing OCR-dependent methods and OCR-free competitors. Code and model are available at https://github.com/whlscut/DocLayLLM.

Problem

• Efficient multi-modal integration for text-rich documents
• Lightweight training for enhanced document comprehension
• Improved OCR information perception in document analysis

Innovation

• Integrates visual and positional tokens into LLMs
• Uses CoT Pre-training and CoT Annealing techniques
• Achieves high performance with lightweight training