Modular Neural Image Signal Processing

2025-12-09

AI Summary

This work addresses three key limitations in neural image signal processing (ISP): limited controllability over intermediate stages, poor cross-camera generalization, and difficulty in style customization. To this end, we propose the first fully controllable, modular neural ISP framework that decomposes the RAW-to-sRGB conversion into interpretable, editable submodules—including white balance, demosaicing, tone mapping, and others—enabling arbitrary intervention and re-rendering at any intermediate stage. The framework employs lightweight, end-to-end trained networks (0.5M–3.9M parameters) and integrates a user-friendly interactive interface for multi-style adaptation and real-time debugging. Extensive evaluation on multiple benchmark datasets demonstrates superior performance over state-of-the-art methods in PSNR, LPIPS, and perceptual quality. Our approach significantly enhances scalability, debuggability, cross-device generalization, and flexibility in adapting to diverse user preferences.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Algorithmic accountability and transparency on the web
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper presents a modular neural image signal processing (ISP) framework that processes raw inputs and renders high-quality display-referred images. Unlike prior neural ISP designs, our method introduces a high degree of modularity, providing full control over multiple intermediate stages of the rendering process.~This modular design not only achieves high rendering accuracy but also improves scalability, debuggability, generalization to unseen cameras, and flexibility to match different user-preference styles. To demonstrate the advantages of this design, we built a user-interactive photo-editing tool that leverages our neural ISP to support diverse editing operations and picture styles. The tool is carefully engineered to take advantage of the high-quality rendering of our neural ISP and to enable unlimited post-editable re-rendering. Our method is a fully learning-based framework with variants of different capacities, all of moderate size (ranging from ~0.5 M to ~3.9 M parameters for the entire pipeline), and consistently delivers competitive qualitative and quantitative results across multiple test sets. Watch the supplemental video at: https://youtu.be/ByhQjQSjxVM

Problem

• Develops a modular neural ISP framework for high-quality image rendering.
• Enhances control, scalability, and generalization in image processing stages.
• Creates an interactive editing tool for flexible post-editable re-rendering.

Innovation

• Modular neural ISP framework processes raw inputs
• High modularity controls multiple intermediate rendering stages
• Learning-based framework supports diverse editing operations