Data-Efficient Adaptation of LLMs via Attention Head Reweighting

2026-07-15

AI Summary

This work addresses the challenge of efficiently adapting large language models to complex text classification tasks under data-scarce conditions. The authors propose Attention Head Reweighting (AHR), a parameter-efficient method that leverages the functional specialization of individual attention heads by learning a single scalar weight per head to reweight its output. Requiring only approximately 0.0001% additional trainable parameters, AHR substantially outperforms LoRA, achieving superior few-shot performance on multiple open-source text classification benchmarks with 200–1000 times fewer trainable parameters. Furthermore, the approach facilitates interpretability by enabling analysis of the contribution of each attention head to the final prediction.

Abstract

Learning effectively from limited data is critical in domains like security where labeled examples are scarce. Large language models (LLMs) have demonstrated some capabilities for data-efficient learning, especially through parameter-efficient adaptation methods, but continue to struggle when faced with few samples for difficult tasks. To meet this challenge, we propose Attention Head Reweighting (AHR), a data-efficient method that adapts LLMs to new text-classification tasks by learning only a single scalar per attention head. This drastically reduces the number of parameters that need to be learned by making use of the functional specialization of individual attention heads. Experiments on diverse open-source text classification datasets show that AHR can outperform standard baselines like LoRA when learning from limited samples, despite having 200-1000x fewer trainable parameters, as our AHR only modifies ~0.0001% of the model's parameters. In addition, our learned weights are easy to interpret and can be analyzed to better understand the mechanisms and attention heads responsible for in-context learning abilities in LLMs.

Problem

• data-efficient learning
• large language models
• few-shot adaptation
• text classification
• limited labeled data

Innovation

• Attention Head Reweighting
• Data-Efficient Adaptation
• Parameter-Efficient Fine-Tuning
• Interpretable Attention
• Large Language Models