Unsupervised Learning of High-resolution Light Field Imaging via Beam Splitter-based Hybrid Lenses

2024-02-29

arXiv.org

AI Summary

Handheld light field (LF) cameras suffer from a fundamental spatial-angular resolution trade-off, resulting in inherently low spatial resolution. Existing supervised LF super-resolution (SR) methods rely on predefined degradation models, leading to significant domain shift between training (natural-resolution LFs as ground truth) and inference (targeting higher-resolution LFs), thus exhibiting poor generalization. Method: We propose the first end-to-end unsupervised LF spatial SR framework. It employs a beam-splitter-based hybrid imaging system that simultaneously captures a 4D LF and a single high-resolution 2D image. Leveraging only this 2D image as supervision, we design a dual pre-trained model-driven loss function that jointly optimizes feature reconstruction and disparity consistency. Contribution/Results: Our method achieves performance on par with state-of-the-art supervised approaches, while being the first to realize fully unsupervised end-to-end learning. We also release the first dedicated hybrid LF dataset, advancing practical deployment of LF SR.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Computational Photography, Image & Video Synthesis
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In this paper, we design a beam splitter-based hybrid light field imaging prototype to record 4D light field image and high-resolution 2D image simultaneously, and make a hybrid light field dataset. The 2D image could be considered as the high-resolution ground truth corresponding to the low-resolution central sub-aperture image of 4D light field image. Subsequently, we propose an unsupervised learning-based super-resolution framework with the hybrid light field dataset, which adaptively settles the light field spatial super-resolution problem with a complex degradation model. Specifically, we design two loss functions based on pre-trained models that enable the super-resolution network to learn the detailed features and light field parallax structure with only one ground truth. Extensive experiments demonstrate the same superiority of our approach with supervised learning-based state-of-the-art ones. To our knowledge, it is the first end-to-end unsupervised learning-based spatial super-resolution approach in light field imaging research, whose input is available from our beam splitter-based hybrid light field system. The hardware and software together may help promote the application of light field super-resolution to a great extent.

Problem

• Self-supervised method for light field spatial super-resolution without predefined degradation models
• Overcoming domain gap between training and inference in real-world data
• Reconstructing high-resolution 4D light fields from low-resolution hand-held captures

Innovation

• Self-supervised learning for light field super-resolution
• Hybrid imaging prototype with central-view reference pairs
• Backward degradation network preserving parallax structures