Personalized Generative Low-light Image Denoising and Enhancement

2024-12-18

arXiv.org

AI Summary

Low-light smartphone imaging is fundamentally limited by photon shot noise and readout noise; existing generative denoising methods often produce content hallucinations under extremely low signal-to-noise ratios (SNR). To address this, we propose a fine-tuning-free personalized generative denoising framework tailored to users’ local photo albums. Our key innovation is the Identity-Consistent Physical Buffer (ICPB) mechanism: it extracts transferable physical attributes—such as facial identity, pose, and skin tone—as strong priors and integrates them into the conditional guidance of diffusion model sampling, enabling end-to-end identity-preserving restoration. By bypassing conventional fine-tuning for personalization, ICPB achieves superior generalization across diverse low-light scenarios. Quantitatively, it improves PSNR by 2.1 dB and SSIM by 0.032 over prior methods; qualitatively, it sets new state-of-the-art performance in visual fidelity and identity consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Privacy

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

While smartphone cameras today can produce astonishingly good photos, their performance in low light is still not completely satisfactory because of the fundamental limits in photon shot noise and sensor read noise. Generative image restoration methods have demonstrated promising results compared to traditional methods, but they suffer from hallucinatory content generation when the signal-to-noise ratio (SNR) is low. Recognizing the availability of personalized photo galleries on users' smartphones, we propose Personalized Generative Denoising (PGD) by building a diffusion model customized for different users. Our core innovation is an identity-consistent physical buffer that extracts the physical attributes of the person from the gallery. This ID-consistent physical buffer provides a strong prior that can be integrated with the diffusion model to restore the degraded images, without the need of fine-tuning. Over a wide range of low-light testing scenarios, we show that PGD achieves superior image denoising and enhancement performance compared to existing diffusion-based denoising approaches.

Problem

• Enhance low-light smartphone photos using personalized generative denoising.
• Reduce hallucinatory content in low SNR generative image restoration.
• Integrate user-specific physical attributes for improved image restoration.

Innovation

• Personalized Generative Denoising (PGD) for low-light images
• ID-consistent physical buffer for user-specific attributes
• Diffusion model integration without fine-tuning