Dive Into the Implicit Biases of Low-rank Vision-language Alignment

2026-07-09

AI Summary

This work addresses the high computational cost and structural disruption of visual features inherent in full-parameter fine-tuning for vision–language alignment. The authors propose aligning only the large language model using low-rank adaptation (LoRA) and systematically investigate the implicit biases it introduces and its impact on model representations. Through geometric analysis, theoretical proofs, and extensive ablation studies encompassing over a hundred configurations and multiple low-rank operators, they demonstrate that low-rank alignment circumvents the “curse of linear separability,” effectively preserves modality-specific knowledge, and enhances the homogeneity, stability, and robustness of visual representations. The method outperforms full-parameter fine-tuning on most benchmarks while substantially reducing computational overhead.

Abstract

Vision-language alignment, the stage that bridges pretrained vision encoders and large language models, is widely treated as a form of pretraining requiring full-parameter updates. We challenge this view and investigate what happens when low-rank adaptation is applied to the LLM during this stage instead. We find that low-rank alignment not only reduces computational costs but also outperforms full-parameter alignment on most benchmarks. To understand this phenomenon, we systematically characterize the implicit biases introduced by low-rank adaptation during alignment. Empirically, we find that low-rank alignment shifts model behavior from hallucinatory to conservative and preserves per-token linear separability of visual features that full-parameter alignment disrupts, a phenomenon we term LS-curse. Geometrically, low rank aligned models exhibit more homogeneous and structurally stable visual representations, maintaining modality-specific knowledge rather than prematurely fusing entity-level semantics. Theoretically, we establish two theorems showing that low-rank alignment induces preferences for parameter subspaces with flat gradients and feature subspaces robust to perturbations, providing a principled explanation for the observed structure-preserving behavior. Extensive experiments cover ablation over 100 alignment configurations, three families of low-rank operators, and various rank, encoder, and other settings.

Problem

• vision-language alignment
• low-rank adaptation
• implicit bias
• parameter efficiency
• representation structure

Innovation

• low-rank adaptation
• vision-language alignment
• implicit bias
• linear separability
• modality-specific representation