All You Need is One: Capsule Prompt Tuning with a Single Vector

2025-10-18

AI Summary

Existing prompt-tuning methods face three key challenges: manual search for optimal prompt length, excessive parameter overhead, and task-aware prompts lacking instance sensitivity—resulting in insufficient attention interaction with input sequences. This paper proposes Capsule Prompt-Tuning (CaPT), introducing the novel “attention anchor” mechanism that encodes both instance semantics and positional information into a single learnable capsule prompt vector, pre-embedded to enhance model focus on input structure. CaPT is end-to-end trainable without requiring prompt-length tuning or redundant prompt tokens. Evaluated on multilingual benchmarks, it achieves significant efficiency and performance gains: 84.03% average accuracy on T5-Large and efficient adaptation of Llama3.2-1B with only 0.003% additional parameters. Its core contribution is the first realization of minimal, jointly task- and instance-aware prompt modeling.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Prompt-based learning has emerged as a parameter-efficient finetuning (PEFT) approach to facilitate Large Language Model (LLM) adaptation to downstream tasks by conditioning generation with task-aware guidance. Despite its successes, current prompt-based learning methods heavily rely on laborious grid searching for optimal prompt length and typically require considerable number of prompts, introducing additional computational burden. Worse yet, our pioneer findings indicate that the task-aware prompt design is inherently limited by its absence of instance-aware information, leading to a subtle attention interplay with the input sequence. In contrast, simply incorporating instance-aware information as a part of the guidance can enhance the prompt-tuned model performance without additional fine-tuning. Moreover, we find an interesting phenomenon, namely "attention anchor", that incorporating instance-aware tokens at the earliest position of the sequence can successfully preserve strong attention to critical structural information and exhibit more active attention interaction with all input tokens. In light of our observation, we introduce Capsule Prompt-Tuning (CaPT), an efficient and effective solution that leverages off-the-shelf, informative instance semantics into prompt-based learning. Our approach innovatively integrates both instance-aware and task-aware information in a nearly parameter-free manner (i.e., one single capsule prompt). Empirical results demonstrate that our method can exhibit superior performance across various language tasks (e.g., 84.03% average accuracy on T5-Large), serving as an "attention anchor," while enjoying high parameter efficiency (e.g., 0.003% of model parameters on Llama3.2-1B).

Problem

• Optimizing prompt length and quantity in prompt-based learning methods
• Addressing absence of instance-aware information in task-aware prompts
• Reducing computational burden while maintaining parameter efficiency

Innovation

• Single capsule prompt integrates instance and task awareness
• Leverages attention anchor for enhanced structural information preservation
• Achieves high performance with minimal parameter usage