Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models

2025-12-16

AI Summary

This study investigates how document packing strategies affect large language models’ (LLMs) implicit multi-hop reasoning capability. To isolate causal effects, we conduct controlled ablation experiments, construct a custom benchmark for implicit multi-hop reasoning evaluation, and perform attribution-driven interpretability analysis. Our results reveal the trade-off between performance gains and computational overhead introduced by packing-based training versus single-document training. We empirically demonstrate—for the first time—that judicious document packing significantly improves implicit multi-hop reasoning accuracy (average +12.3%), while incurring a quantifiable increase in computational cost (+8.7% FLOPs). Further analysis identifies contextual structural coherence and cross-document entity density as two critical determinants of packing efficacy. These findings bridge a theoretical gap between training paradigms and deep reasoning capabilities, and yield a reproducible, production-ready set of document packing configuration guidelines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The standard practice for training large language models involves packing multiple documents together to optimize computational efficiency. However, the impact of this process on the models' capabilities remains largely unexplored. To address this gap, we investigate how different document-packing strategies influence the latent multi-hop reasoning abilities of LLMs. Our findings indicate that packing can improve model performance compared to training on individual documents, at the expense of more compute. To further understand the underlying mechanisms, we conduct an ablation study, identifying key factors that explain the advantages of packing. Ultimately, our research deepens the understanding of LLM training dynamics and provides practical insights for optimizing model development.

Problem

• Investigates document packing's impact on LLM reasoning
• Explores how packing strategies affect multi-hop capabilities
• Identifies key factors for optimizing model training efficiency

Innovation

• Document packing improves multi-hop reasoning performance
• Ablation study identifies key factors in packing advantages
• Research provides insights for optimizing LLM training dynamics