Beyond Subspace Isolation: Many-to-Many Transformer for Light Field Image Super-resolution

2024-01-01

IEEE transactions on multimedia

AI Summary

Existing light field super-resolution methods decompose 4D data into lower-dimensional subspaces for separate processing, restricting self-attention to local sub-apertures and impeding global spatial-angular dependency modeling—a limitation we term “subspace isolation.” To address this, we propose the Many-to-Many Transformer (M2MT), which first aggregates angular information in the spatial domain and then applies full-sub-aperture many-to-many self-attention, breaking the conventional one-to-one local constraint to enable long-range joint spatial-angular modeling. Our method integrates a lightweight spatial feature aggregation module with an enhanced Transformer architecture and introduces a Local Attribution Map (LAM) to improve interpretability. Evaluated on multiple benchmark datasets, M2MT achieves state-of-the-art performance. Ablation and LAM analysis confirm its capability to effectively capture cross-dimensional global correlations, significantly improving reconstruction accuracy and structural fidelity.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The effective extraction of spatial-angular features plays a crucial role in light field image super-resolution (LFSR) tasks, and the introduction of convolution and Transformers leads to significant improvement in this area. Nevertheless, due to the large 4D data volume of light field images, many existing methods opted to decompose the data into a number of lower-dimensional subspaces and perform Transformers in each sub-space individually. As a side effect, these methods inadvertently restrict the self-attention mechanisms to a One-to-One scheme accessing only a limited subset of LF data, explicitly preventing comprehensive optimization on all spatial and angular cues. In this paper, we identify this limitation as subspace isolation and introduce a novel Many-to-Many Transformer (M2MT) to address it. M2MT aggregates angular information in the spatial subspace before performing the self-attention mechanism. It enables complete access to all information across all sub-aperture images (SAIs) in a light field image. Consequently, M2MT is enabled to comprehensively capture long-range correlation dependencies. With M2MT as the pivotal component, we develop a simple yet effective M2MT network for LFSR. Our experimental results demonstrate that M2MT achieves state-of-the-art performance across various public datasets. We further conduct in-depth analysis using local attribution maps (LAM) to obtain visual interpretability, and the results validate that M2MT is empowered with a truly non-local context in both spatial and angular subspaces to mitigate subspace isolation and acquire effective spatial-angular representation.

Problem

• Addresses subspace isolation in light field image super-resolution.
• Introduces Many-to-Many Transformer for comprehensive spatial-angular feature extraction.
• Enables full access to all sub-aperture images for better optimization.

Innovation

• Many-to-Many Transformer for comprehensive data access
• Aggregates angular information before self-attention mechanism
• Achieves state-of-the-art light field image super-resolution