Expanding before Inferring: Enhancing Factuality in Large Language Models through Premature Layers Interpolation

2025-06-03

AI Summary

Large language models (LLMs) suffer from pervasive factual hallucinations. Existing mitigation strategies—such as fine-tuning, retrieval augmentation, or post-hoc correction—often incur substantial computational overhead and overlook the critical role of early transformer layers in refining factual representations. This paper proposes PLI (Early-Layer Interpolation), a training-free, plug-and-play intervention method that performs linear or nonlinear interpolation between “precocious” early layers of the transformer. PLI further incorporates a diffusion-inspired depth-expansion mechanism to elongate the representation processing path and enhance factual consistency. To our knowledge, PLI is the first approach to apply inter-layer interpolation specifically within early layers to improve factual accuracy, without relying on external knowledge sources or parameter updates. Extensive experiments demonstrate that PLI significantly reduces hallucination rates across four mainstream factual evaluation benchmarks, consistently outperforming retrieval-augmented generation, post-hoc correction, and alignment-focused fine-tuning baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) demonstrate remarkable capabilities in text understanding and generation. However, their tendency to produce factually inconsistent outputs, commonly referred to as ''hallucinations'', remains a critical challenge. Existing approaches, such as retrieval-based and inference-time correction methods, primarily address this issue at the input or output level, often overlooking the intrinsic information refinement process and the role of premature layers. Meanwhile, alignment- and fine-tuning-based methods are resource-intensive. In this paper, we propose PLI (Premature Layers Interpolation), a novel, training-free, and plug-and-play intervention designed to enhance factuality. PLI mitigates hallucinations by inserting premature layers formed through mathematical interpolation with adjacent layers. Inspired by stable diffusion and sampling steps, PLI extends the depth of information processing and transmission in LLMs, improving factual coherence. Experiments on four publicly available datasets demonstrate that PLI effectively reduces hallucinations while outperforming existing baselines in most cases. Further analysis suggests that the success of layer interpolation is closely linked to LLMs' internal mechanisms. To promote reproducibility, we will release our code and data upon acceptance.

Problem

• Reducing factual inconsistencies in Large Language Models
• Addressing hallucinations without resource-intensive fine-tuning
• Enhancing factuality through premature layer interpolation

Innovation

• Premature Layers Interpolation (PLI) enhances factuality
• Training-free plug-and-play intervention reduces hallucinations
• Mathematical interpolation extends information processing depth