Not All Directions Matter: Toward Structured and Task-Aware Low-Rank Adaptation

2026-03-15

AI Summary

This work proposes StructLoRA, a novel parameter-efficient fine-tuning method that jointly addresses semantic drift and inter-layer structural inconsistency—two key limitations of Low-Rank Adaptation (LoRA). StructLoRA introduces an information bottleneck-guided filter to suppress task-irrelevant directions and a lightweight graph coordinator to align layer-wise updates during training. The approach incurs no additional inference overhead and achieves state-of-the-art performance across diverse architectures, including LLaMA, LLaVA, and ViT. Notably, StructLoRA demonstrates substantial improvements over existing methods under low-rank and few-shot settings, where maintaining semantic fidelity and structural coherence is particularly challenging.

Technology Category

• Machine Learning: Structured Learning
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT). Yet, its efficacy is hampered by two fundamental limitations: semantic drift, by treating all update directions with equal importance, and structural incoherence, from adapting layers independently, resulting in suboptimal, uncoordinated updates. To remedy these, we propose StructLoRA, a framework that addresses both limitations through a principled, dual-component design: (1) an Information Bottleneck-guided filter that prunes task-irrelevant directions to mitigate semantic drift, and (2) a lightweight, training-only graph-based coordinator that enforces inter-layer consistency to resolve structural incoherence. Extensive experiments across large language model , vision language model, and vision model (including LLaMA, LLaVA, and ViT) demonstrate that StructLoRA consistently establishes a new state-of-the-art, outperforming not only vanilla LoRA but also advanced dynamic rank allocation and sparsity-based methods. Notably, the benefits are particularly pronounced in challenging low-rank and low-data regimes. Crucially, since our proposed modules operate only during training, StructLoRA enhances performance with zero additional inference cost, advancing the focus of PEFT -- from mere parameter compression to a more holistic optimization of information quality and structural integrity.

Problem

• Low-Rank Adaptation
• semantic drift
• structural incoherence
• parameter-efficient fine-tuning
• task-aware adaptation

Innovation

• Low-Rank Adaptation
• Information Bottleneck
• Structural Consistency
• Parameter-Efficient Fine-Tuning
• Graph-based Coordination