Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs

2025-06-05

AI Summary

To address the high computational cost and technical barriers associated with fine-tuning large language models (LLMs), this paper proposes an input-dependent soft prompt method. Our approach leverages self-attention to dynamically weight input tokens and generate learnable soft prompt embeddings conditioned on the current input’s semantic content, enabling lightweight, conditional parameter adaptation. Unlike static or task-level fixed prompts, ours is the first to formulate prompt generation as an input-conditioned process. With negligible parameter overhead (<0.1%), it significantly improves zero-shot cross-domain transfer performance. Extensive experiments across multiple downstream tasks demonstrate consistent superiority over mainstream soft-prompting baselines—including Prefix-Tuning and Prompt-Tuning—validating the synergistic gains from dynamically aligning prompt relevance with input semantics and enhancing generalization.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.

Problem

• Efficient fine-tuning for domain-specific LLM tasks
• Input-dependent soft prompting with self-attention
• Improving zero-shot domain transfer capability

Innovation

• Input-dependent soft prompting with self-attention
• Efficient fine-tuning with minimal parameters
• Improved zero-shot domain transfer capability