GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning

2025-11-15

AI Summary

Existing parameter-efficient fine-tuning (PEFT) methods apply static, input-agnostic updates to all tokens, leading to overfitting on simple samples or insufficient adaptation at critical token positions. To address this, we propose GateRA—a dynamic, input-aware gating framework for PEFT—that integrates soft gating mechanisms into LoRA-like architectures to enable token-level adaptive parameter updates. Theoretically, GateRA induces a soft gradient masking effect; additionally, entropy regularization is introduced to sparsify the gating distribution, enhancing interpretability and generalization. Empirically, GateRA achieves significant improvements or matches state-of-the-art PEFT methods across multiple commonsense reasoning benchmarks. Visualization analyses confirm that GateRA suppresses redundant updates during the prefill phase while strengthening modeling capacity for salient tokens during autoregressive decoding.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Generation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, DoRA, and HiRA, enable lightweight adaptation of large pre-trained models via low-rank updates. However, existing PEFT approaches apply static, input-agnostic updates to all tokens, disregarding the varying importance and difficulty of different inputs. This uniform treatment can lead to overfitting on trivial content or under-adaptation on more informative regions, especially in autoregressive settings with distinct prefill and decoding dynamics. In this paper, we propose GateRA, a unified framework that introduces token-aware modulation to dynamically adjust the strength of PEFT updates. By incorporating adaptive gating into standard PEFT branches, GateRA enables selective, token-level adaptation, preserving pre-trained knowledge for well-modeled inputs while focusing capacity on challenging cases. Empirical visualizations reveal phase-sensitive behaviors, where GateRA automatically suppresses updates for redundant prefill tokens while emphasizing adaptation during decoding. To promote confident and efficient modulation, we further introduce an entropy-based regularization that encourages near-binary gating decisions. This regularization prevents diffuse update patterns and leads to interpretable, sparse adaptation without hard thresholding. Finally, we present a theoretical analysis showing that GateRA induces a soft gradient-masking effect over the PEFT path, enabling continuous and differentiable control over adaptation. Experiments on multiple commonsense reasoning benchmarks demonstrate that GateRA consistently outperforms or matches prior PEFT methods.

Problem

• Dynamically adjusts PEFT updates based on token importance
• Prevents overfitting on trivial content and under-adaptation on critical regions
• Enables selective token-level adaptation while preserving pre-trained knowledge

Innovation

• Token-aware modulation for dynamic PEFT updates
• Entropy regularization for near-binary gating decisions
• Soft gradient-masking effect enabling differentiable adaptation