Learned Lightweight Smartphone ISP with Unpaired Data

2025-05-15

AI Summary

This work addresses the fundamental limitation of deep learning-based image signal processing (ISP) models—reliance on pixel-aligned RAW–RGB paired data—by proposing the first learnable, unpaired ISP framework tailored for mobile deployment. Methodologically, we design a lightweight neural architecture integrated with multi-scale discriminators and perception-guided losses leveraging pre-trained VGG/ResNet features, enabling unsupervised adversarial training that preserves structural integrity and faithfully reconstructs color and texture without image registration. Evaluated on Zurich RAW2RGB and Fujifilm UltraISP benchmarks, our method achieves PSNR, SSIM, and LPIPS scores competitive with fully supervised approaches, while maintaining low parameter count and high inference efficiency—enabling edge-device deployment. Our core contributions are: (1) the first unpaired RAW-to-RGB translation paradigm for ISP; and (2) a practical, lightweight solution that jointly optimizes performance, model compactness, and deployability.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The Image Signal Processor (ISP) is a fundamental component in modern smartphone cameras responsible for conversion of RAW sensor image data to RGB images with a strong focus on perceptual quality. Recent work highlights the potential of deep learning approaches and their ability to capture details with a quality increasingly close to that of professional cameras. A difficult and costly step when developing a learned ISP is the acquisition of pixel-wise aligned paired data that maps the raw captured by a smartphone camera sensor to high-quality reference images. In this work, we address this challenge by proposing a novel training method for a learnable ISP that eliminates the need for direct correspondences between raw images and ground-truth data with matching content. Our unpaired approach employs a multi-term loss function guided by adversarial training with multiple discriminators processing feature maps from pre-trained networks to maintain content structure while learning color and texture characteristics from the target RGB dataset. Using lightweight neural network architectures suitable for mobile devices as backbones, we evaluated our method on the Zurich RAW to RGB and Fujifilm UltraISP datasets. Compared to paired training methods, our unpaired learning strategy shows strong potential and achieves high fidelity across multiple evaluation metrics. The code and pre-trained models are available at https://github.com/AndreiiArhire/Learned-Lightweight-Smartphone-ISP-with-Unpaired-Data .

Problem

• Develops lightweight smartphone ISP without paired training data
• Uses adversarial training for color and texture learning
• Achieves high fidelity with unpaired learning strategy

Innovation

• Unpaired training method for learned ISP
• Multi-term loss with adversarial discriminators
• Lightweight neural network for mobile devices