Experts-Guided Unbalanced Optimal Transport for ISP Learning from Unpaired and/or Paired Data

2025-12-05

AI Summary

Existing learning-based image signal processing (ISP) methods heavily rely on large-scale paired raw-sRGB datasets, which are expensive to acquire and limit generalization. Method: We propose the first unsupervised ISP training framework grounded in unbalanced optimal transport (UOT), enabling joint training on both paired and unpaired data. Our approach innovatively integrates UOT into ISP, coupled with an ensemble expert discriminator and hybrid adversarial regularization to jointly suppress interference from out-of-distribution samples in the target domain and specifically correct color shifts, structural distortions, and frequency-domain artifacts. Contribution/Results: Experiments demonstrate state-of-the-art performance under paired settings and competitive—often superior—performance under unpaired settings compared to leading paired methods. This work substantially reduces data dependency while enhancing the practicality, robustness, and cross-domain generalization capability of ISP models.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Unsupervised & Self-Supervised Learning

Application Category

• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Learned Image Signal Processing (ISP) pipelines offer powerful end-to-end performance but are critically dependent on large-scale paired raw-to-sRGB datasets. This reliance on costly-to-acquire paired data remains a significant bottleneck. To address this challenge, we introduce a novel, unsupervised training framework based on Optimal Transport capable of training arbitrary ISP architectures in both unpaired and paired modes. We are the first to successfully apply Unbalanced Optimal Transport (UOT) for this complex, cross-domain translation task. Our UOT-based framework provides robustness to outliers in the target sRGB data, allowing it to discount atypical samples that would be prohibitively costly to map. A key component of our framework is a novel ``committee of expert discriminators,'' a hybrid adversarial regularizer. This committee guides the optimal transport mapping by providing specialized, targeted gradients to correct specific ISP failure modes, including color fidelity, structural artifacts, and frequency-domain realism. To demonstrate the superiority of our approach, we retrained existing state-of-the-art ISP architectures using our paired and unpaired setups. Our experiments show that while our framework, when trained in paired mode, exceeds the performance of the original paired methods across all metrics, our unpaired mode concurrently achieves quantitative and qualitative performance that rivals, and in some cases surpasses, the original paired-trained counterparts. The code and pre-trained models are available at: https://github.com/gosha20777/EGUOT-ISP.git.

Problem

• Reducing dependency on costly paired raw-to-sRGB datasets for ISP training
• Enabling ISP learning from both unpaired and paired data using Optimal Transport
• Correcting specific ISP failures like color fidelity and structural artifacts

Innovation

• Unbalanced Optimal Transport for unpaired ISP training
• Committee of expert discriminators guides transport mapping
• Robust framework works with both paired and unpaired data