Deep Pre-Alignment for VLMs

2026-05-14

AI Summary

This work addresses the limitation in current vision-language models (VLMs), where insufficient alignment between visual and textual representation spaces in early layers consumes substantial network depth for shallow modality alignment, thereby constraining deeper reasoning capabilities. To overcome this, the authors propose Deep Pre-Alignment (DPA), an architecture that offloads modality alignment to a lightweight, small-scale VLM encoder, achieving deep pre-alignment prior to feeding inputs into large language models such as Qwen3 or LLaMA 3.2. This design frees the main LLM backbone to focus on high-level reasoning and integrates seamlessly into existing VLM frameworks. Experiments demonstrate that DPA yields average multimodal performance gains of 1.9 and 3.0 points at 4B and 32B scales, respectively, while reducing language capability forgetting by 32.9%.

Abstract

Most Vision Language Models (VLMs) directly map outputs from ViT encoders to the LLM via a lightweight projector. While effective, recent analysis suggests this architecture suffers from an alignment challenge: visual features remain distant from the text space in the initial layers of the LLM, forcing the model to waste critical depth~\cite{zhang-etal-2024-investigating,artzy-schwartz-2024-attend} on superficial modality alignment rather than deep understanding and complex reasoning. In this work, we propose Deep Pre-Alignment (DPA), a novel architecture that replaces the standard ViT encoder with a small VLM as perceiver, ensuring visual features are deeply aligned with the text space of the target large language model. Comprehensive experiments demonstrate the effectiveness of DPA. On the 4B parameter scale, DPA outperforms baselines by 1.9 points across 8 multimodal benchmarks, with gains widening to 3.0 points at the 32B scale. Moreover, by offloading alignment to the perceiver, DPA achieves a 32.9\% reduction in language capability forgetting over 3 text benchmarks. We further demonstrate that these gains are consistent across different LLM families including Qwen3 and LLaMA 3.2, highlighting the generality of our approach. Beyond performance, DPA also offers a seamless upgrade path for current VLM development, requiring only a modular replacement for the visual encoder with marginal computation overhead.

Problem

• Vision Language Models
• modality alignment
• visual-text alignment
• LLM integration
• feature space discrepancy

Innovation

• Deep Pre-Alignment
• Vision Language Models
• Modality Alignment
• Perceiver Architecture
• Language Capability Preservation