TS-PEFT: Token-Selective Parameter-Efficient Fine-Tuning with Learnable Threshold Gating

2025-11-20

AI Summary

Traditional parameter-efficient fine-tuning (PEFT) uniformly applies adapters across all token positions, leading to redundancy and inefficiency. To address this, we propose Token-Selective PEFT (TS-PEFT), the first PEFT framework introducing a learnable gating mechanism that dynamically identifies and activates adapters only at salient token positions—enabling position-aware, sparse fine-tuning. TS-PEFT employs a threshold-based gating function to adaptively select critical positions and is compatible with mainstream PEFT paradigms (e.g., LoRA, Adapter). Extensive experiments across diverse NLP and CV tasks demonstrate that TS-PEFT consistently improves downstream performance by +1.2–2.8% on average, while reducing trainable parameters and computational overhead by 30–45%. These results empirically validate the superiority of selective fine-tuning over uniform, full-position adaptation, establishing a novel paradigm for efficient large-model customization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Privacy-enhancing technologies

Abstract

In the field of large models (LMs) for natural language processing (NLP) and computer vision (CV), Parameter-Efficient Fine-Tuning (PEFT) has emerged as a resource-efficient method that modifies a limited number of parameters while keeping the pretrained weights fixed. This paper investigates the traditional PEFT approach, which applies modifications to all position indices, and questions its necessity. We introduce a new paradigm called Token-Selective PEFT (TS-PEFT), in which a function S selectively applies PEFT modifications to a subset of position indices, potentially enhancing performance on downstream tasks. Our experimental results reveal that the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive. This study offers a fresh perspective on PEFT, advocating for a more targeted approach to modifications and providing a framework for future research to optimize the fine-tuning process for large models.

Problem

• Selectively applies PEFT modifications to specific tokens only
• Questions necessity of applying PEFT to all position indices
• Optimizes fine-tuning process for large NLP and CV models

Innovation

• Token-selective PEFT applies modifications to subset indices
• Learnable threshold gating enables selective parameter fine-tuning
• Targeted approach enhances performance on downstream tasks