Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models

2026-02-22

AI Summary

This work proposes Astra, a novel parameter-efficient fine-tuning (PEFT) method that addresses a key limitation of existing low-rank adaptation approaches such as LoRA: their neglect of the subspace spanned by tail eigenvectors in the activation space, which constrains fine-tuning performance. Astra is the first to systematically leverage a task-specific calibration set to estimate these tail eigenvectors of output activations, constructing a task-adaptive low-rank adapter whose parameter updates are constrained within this informative subspace. By identifying critical activation directions via eigendecomposition, Astra achieves significantly faster convergence and improved model performance with only a marginal increase in trainable parameters. Evaluated across 16 NLU and NLG benchmarks, Astra consistently outperforms state-of-the-art PEFT methods and, in several tasks, even surpasses full fine-tuning.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, are widely used for adapting pre-trained models to downstream tasks due to their computational and storage efficiency. However, in the context of LoRA and its variants, the potential of activation subspaces corresponding to tail eigenvectors remains substantially under-exploited, which may lead to suboptimal fine-tuning performance. In this work, we propose Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation), a novel PEFT method that leverages the tail eigenvectors of the model output activations-estimated from a small task-specific calibration set-to construct task-adaptive low-rank adapters. By constraining updates to the subspace spanned by these tail eigenvectors, Astra achieves faster convergence and improved downstream performance with a significantly reduced parameter budget. Extensive experiments across natural language understanding (NLU) and natural language generation (NLG) tasks demonstrate that Astra consistently outperforms existing PEFT baselines across 16 benchmarks and even surpasses full fine-tuning (FFT) in certain scenarios.

Problem

• Parameter-Efficient Fine-Tuning
• LoRA
• activation subspace
• tail eigenvectors
• large language models

Innovation

• Parameter-Efficient Fine-Tuning
• LoRA
• Tail Eigenvectors
• Activation Subspace
• Low-Rank Adaptation