Exploring Scalable Unified Modeling for General Low-Level Vision

2025-07-19

AI Summary

Low-level vision tasks—such as image restoration, enhancement, stylization, and feature extraction—exhibit substantial heterogeneity in task definitions and output domains, hindering unified modeling. To address this, we propose the Vision Prompt-based Image Processing framework (VPIP), the first to introduce visual prompting into low-level vision. VPIP employs input–target image pairs as task-specific prompts to steer a scalable, end-to-end general-purpose model, GenLV. Its architecture comprises a prompt encoder, a prompt interaction module, and a plug-and-play backbone network, enabling zero-shot generalization and few-shot transfer. Trained jointly across a large-scale, multi-task dataset, VPIP demonstrates effectiveness on a benchmark spanning over 100 diverse low-level vision tasks. Experimental results show that both model scaling and task diversity expansion consistently improve performance, significantly enhancing cross-task adaptability and generalization under data-scarce conditions.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Privacy-enhancing technologies

Abstract

Low-level vision involves a wide spectrum of tasks, including image restoration, enhancement, stylization, and feature extraction, which differ significantly in both task formulation and output domains. To address the challenge of unified modeling across such diverse tasks, we propose a Visual task Prompt-based Image Processing (VPIP) framework that leverages input-target image pairs as visual prompts to guide the model in performing a variety of low-level vision tasks. The framework comprises an end-to-end image processing backbone, a prompt encoder, and a prompt interaction module, enabling flexible integration with various architectures and effective utilization of task-specific visual representations. Based on this design, we develop a unified low-level vision model, GenLV, and evaluate its performance across multiple representative tasks. To explore the scalability of this approach, we extend the framework along two dimensions: model capacity and task diversity. We construct a large-scale benchmark consisting of over 100 low-level vision tasks and train multiple versions of the model with varying scales. Experimental results show that the proposed method achieves considerable performance across a wide range of tasks. Notably, increasing the number of training tasks enhances generalization, particularly for tasks with limited data, indicating the model's ability to learn transferable representations through joint training. Further evaluations in zero-shot generalization, few-shot transfer, and task-specific fine-tuning scenarios demonstrate the model's strong adaptability, confirming the effectiveness, scalability, and potential of the proposed framework as a unified foundation for general low-level vision modeling.

Problem

• Unified modeling for diverse low-level vision tasks
• Scalable framework for general image processing
• Enhancing generalization across 100+ vision tasks

Innovation

• Visual task Prompt-based Image Processing framework
• End-to-end backbone with prompt encoder
• Scalable unified model for diverse tasks