Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction

2025-06-05

AI Summary

Efficient, lightweight downstream adaptation of large language models remains challenging due to fragmented methodologies and lack of unifying principles. Method: This paper proposes a unified framework—“neural reprogrammability”—modeling parameter-free adaptation paradigms—including model reprogramming, prompt tuning, and prompt instruction—as targeted manipulations of information flow at interfaces such as input, intermediate layers, or context. Contribution/Results: We introduce the first cross-modal, architecture-agnostic four-dimensional taxonomy (format, location, operator, output alignment), revealing intrinsic unity among in-context learning, chain-of-thought, and related methods. By systematically integrating existing interface perturbation techniques—including input perturbation, token insertion, and example injection—we empirically validate their generality across multimodal foundation models. Our framework establishes foundational principles and provides actionable guidelines for lightweight, controllable, and interpretable model adaptation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

As large-scale pre-trained foundation models continue to expand in size and capability, efficiently adapting them to specific downstream tasks has become increasingly critical. Despite substantial progress, existing adaptation approaches have evolved largely in isolation, without a clear understanding of their interrelationships. This survey introduces neural network reprogrammability as a unifying framework that bridges mainstream model adaptation techniques--model reprogramming, prompt tuning, and prompt instruction--previously fragmented research areas yet converges on a shared principle: repurposing a pre-trained model by manipulating information at the interfaces while keeping the model parameters frozen. These methods exploit neural networks' sensitivity to manipulation on different interfaces, be it through perturbing inputs, inserting tokens into intermediate layers, or providing task-specific examples in context, to redirect model behaviors towards desired outcomes. We then present a taxonomy that categorizes such information manipulation-based adaptation approaches across four key dimensions: manipulation format (fixed or learnable), location (interfaces where manipulations occur), operator (how they are applied), and output alignment requirement (post-processing needed to align outputs with downstream tasks). Notably, this framework applies consistently across data modalities, independent of specific model architectures. Moreover, viewing established techniques like in-context learning and chain-of-thought prompting through this lens reveals both their theoretical connections and practical distinctions. We further analyze remaining technical challenges and ethical considerations, positioning neural network reprogrammability as a fundamental paradigm for efficient model adaptation. We lastly identify promising research directions emerging from this integrative viewpoint.

Problem

• Unifying model adaptation techniques for pre-trained foundation models
• Exploring neural network sensitivity to interface manipulations
• Categorizing adaptation approaches across key dimensions systematically

Innovation

• Unified framework for model adaptation techniques
• Manipulate interfaces with frozen model parameters
• Taxonomy categorizes adaptation across four dimensions