TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal

2025-08-11

AI Summary

To address parameter redundancy and insufficient task correlation modeling in multi-task weather image restoration, this paper proposes a parameter-efficient unified framework. The method introduces a task-aware low-rank soft prompt mechanism, where learnable prompt embeddings jointly represent diverse weather degradations; incorporates contrastive constraints to explicitly model inter-task semantic correlations; and adopts a two-stage training strategy—supervised pre-training followed by prompt-based fine-tuning—to synergistically integrate supervised and contrastive learning. With only 2.75 million parameters, the approach achieves state-of-the-art performance across multiple weather degradation tasks. t-SNE visualization confirms effective alignment of task-specific representations, demonstrating significant improvements in generalization capability and multi-task collaborative representation learning.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Image restoration under adverse weather conditions has been extensively explored, leading to numerous high-performance methods. In particular, recent advances in All-in-One approaches have shown impressive results by training on multi-task image restoration datasets. However, most of these methods rely on dedicated network modules or parameters for each specific degradation type, resulting in a significant parameter overhead. Moreover, the relatedness across different restoration tasks is often overlooked. In light of these issues, we propose a parameter-efficient All-in-One image restoration framework that leverages task-aware enhanced prompts to tackle various adverse weather degradations.Specifically, we adopt a two-stage training paradigm consisting of a pretraining phase and a prompt-tuning phase to mitigate parameter conflicts across tasks. We first employ supervised learning to acquire general restoration knowledge, and then adapt the model to handle specific degradation via trainable soft prompts. Crucially, we enhance these task-specific prompts in a task-aware manner. We apply low-rank decomposition to these prompts to capture both task-general and task-specific characteristics, and impose contrastive constraints to better align them with the actual inter-task relatedness. These enhanced prompts not only improve the parameter efficiency of the restoration model but also enable more accurate task modeling, as evidenced by t-SNE analysis. Experimental results on different restoration tasks demonstrate that the proposed method achieves superior performance with only 2.75M parameters.

Problem

• Reducing parameter overhead in adverse weather image restoration
• Enhancing task-relatedness modeling across restoration tasks
• Improving parameter efficiency with task-aware prompting

Innovation

• Parameter-efficient task-aware prompting for weather removal
• Two-stage training with pretraining and prompt-tuning
• Low-rank decomposition for task-general and task-specific prompts