Iterative Approach to Reconstructing Neural Disparity Fields From Light-Field Data

2024-07-22

IEEE Transactions on Computational Imaging

AI Summary

Traditional discrete disparity maps suffer from sampling errors and interpolation artifacts, limiting their ability to accurately represent continuous disparity variations. To address this, we propose an unsupervised, differentiable iterative optimization framework that enables the first light-field-data-driven reconstruction of neural disparity fields (NDFs). Our method integrates a physics-based differentiable light-field forward model with spatial-angular consistency constraints, and employs a hash-encoded MLP to represent disparity as an implicit continuous function—requiring no ground-truth annotations or pre-trained models. The framework is agnostic to light-field acquisition configurations and supports sub-pixel disparity sampling and rendering at arbitrary resolutions. Experiments demonstrate substantial improvements in disparity reconstruction fidelity and generalization across complex textures and geometries, effectively overcoming fundamental limitations of conventional disparity maps in both accuracy and continuity.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Computer Vision: Learning & Optimization for CV
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

This study proposes a neural disparity field (NDF) that establishes an implicit, continuous representation of scene disparity based on a neural field and an iterative approach to address the inverse problem of NDF reconstruction from light-field (LF) data. NDF enables seamless and precise characterization of disparity variations in three-dimensional scenes and can discretize disparity at any arbitrary resolution, overcoming the limitations of traditional disparity maps that are prone to sampling errors and interpolation inaccuracies. The proposed NDF network architecture utilizes hash encoding combined with multilayer perceptrons (MLPs) to capture detailed disparities in texture levels, thereby enhancing its ability to represent the geometric information of complex scenes. By leveraging the spatial-angular consistency inherent in the LF data, a differentiable forward model to generate a central view image from the LF data is developed. Based on the forward model, an optimization scheme for the inverse problem of NDF reconstruction using differentiable propagation operators is established. Furthermore, an iterative solution method is adopted to reconstruct the NDF in the optimization scheme, which does not require training datasets and applies to LF data captured by various acquisition methods. Experimental results demonstrate that the proposed method can reconstruct high-quality NDF from LF data. The high-resolution disparity can be effectively recovered by NDF, demonstrating its capability for the implicit, continuous representation of scene disparities.

Problem

• Reconstructs neural disparity fields from light-field data iteratively
• Overcomes sampling errors in traditional disparity map methods
• Enables continuous high-resolution disparity without training datasets

Innovation

• Neural disparity field for continuous scene representation
• Hash encoding with MLPs for detailed disparity capture
• Differentiable forward model for light-field data optimization