Task-Aware Image Signal Processor for Advanced Visual Perception

2025-09-17

AI Summary

Existing RAW-domain vision methods face two key bottlenecks: end-to-end ISP networks suffer from excessive parameter count and high computational cost, while traditional ISP pipeline tuning is fundamentally limited in representational capacity. This paper proposes a lightweight task-oriented RAW-to-RGB framework, whose core innovation is a decomposed multi-scale modulation operator. This operator employs lightweight convolutions to generate globally coherent, region-aware, and pixel-level statistical control signals—directly reshaping feature distributions within the RAW domain. Enabled by minimal parameters (<0.1M), it achieves spatially variant, cross-scale transformations, overcoming inherent representational constraints of conventional ISP pipelines. Evaluated on day/night multi-scene object detection and semantic segmentation, our method consistently improves accuracy (+2.3–4.1 mAP/mIoU) while reducing model parameters by 72% and inference latency by 58%, enabling efficient deployment on edge devices.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In recent years, there has been a growing trend in computer vision towards exploiting RAW sensor data, which preserves richer information compared to conventional low-bit RGB images. Early studies mainly focused on enhancing visual quality, while more recent efforts aim to leverage the abundant information in RAW data to improve the performance of visual perception tasks such as object detection and segmentation. However, existing approaches still face two key limitations: large-scale ISP networks impose heavy computational overhead, while methods based on tuning traditional ISP pipelines are restricted by limited representational capacity.To address these issues, we propose Task-Aware Image Signal Processing (TA-ISP), a compact RAW-to-RGB framework that produces task-oriented representations for pretrained vision models. Instead of heavy dense convolutional pipelines, TA-ISP predicts a small set of lightweight, multi-scale modulation operators that act at global, regional, and pixel scales to reshape image statistics across different spatial extents. This factorized control significantly expands the range of spatially varying transforms that can be represented while keeping memory usage, computation, and latency tightly constrained. Evaluated on several RAW-domain detection and segmentation benchmarks under both daytime and nighttime conditions, TA-ISP consistently improves downstream accuracy while markedly reducing parameter count and inference time, making it well suited for deployment on resource-constrained devices.

Problem

• Optimizing RAW-to-RGB conversion for visual perception tasks
• Reducing computational overhead in image signal processing
• Enhancing task performance while maintaining resource efficiency

Innovation

• Lightweight multi-scale modulation operators
• Factorized control for varying spatial transforms
• Task-oriented RAW-to-RGB framework optimization