Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention

2026-02-06

AI Summary

This work addresses the high computational complexity and rigid parameter sharing inherent in existing full self-attention-based novel view synthesis methods. To overcome these limitations, we propose a dual-stream Transformer architecture that decouples the processing of input and target views through a collaborative refinement mechanism: self-attention is applied to the input views, while self-post-cross attention operates on the target views. Furthermore, we introduce an incremental inference strategy compatible with KV caching to significantly reduce redundant computation. Our method achieves a PSNR of 29.86 dB on RealEstate10K using only two input views, outperforming LVSM by 0.2 dB, while accelerating training convergence by 2× and inference by 4.4×. It attains state-of-the-art performance across multiple metrics and demonstrates zero-shot generalization to unseen numbers of input views.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Feedforward models for novel view synthesis (NVS) have recently advanced by transformer-based methods like LVSM, using attention among all input and target views. In this work, we argue that its full self-attention design is suboptimal, suffering from quadratic complexity with respect to the number of input views and rigid parameter sharing among heterogeneous tokens. We propose Efficient-LVSM, a dual-stream architecture that avoids these issues with a decoupled co-refinement mechanism. It applies intra-view self-attention for input views and self-then-cross attention for target views, eliminating unnecessary computation. Efficient-LVSM achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed. Efficient-LVSM achieves state-of-the-art performance on multiple benchmarks, exhibits strong zero-shot generalization to unseen view counts, and enables incremental inference with KV-cache, thanks to its decoupled designs.

Problem

• novel view synthesis
• self-attention
• computational complexity
• parameter sharing
• transformer

Innovation

• Efficient-LVSM
• decoupled co-refinement
• novel view synthesis
• transformer attention
• incremental inference