NVS-HO: A Benchmark for Novel View Synthesis of Handheld Objects

2026-02-05

AI Summary

This work addresses the challenge of novel view synthesis for hand-held objects in real-world scenes using only RGB inputs. It introduces the first benchmark specifically designed for this task, comprising hand-held object sequences for training and checkerboard-based calibration sequences that provide ground-truth camera poses for evaluation. To tackle the problem, the authors estimate camera poses by combining Structure-from-Motion (SfM) with a pre-trained VGG network and develop a synthesis model that integrates Neural Radiance Fields (NeRF) with 3D Gaussian splatting. Experimental results demonstrate that existing methods suffer significant performance degradation under the non-rigid, unconstrained conditions typical of hand-held objects, thereby highlighting the critical role of this benchmark in advancing robust novel view synthesis research.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

We propose NVS-HO, the first benchmark designed for novel view synthesis of handheld objects in real-world environments using only RGB inputs. Each object is recorded in two complementary RGB sequences: (1) a handheld sequence, where the object is manipulated in front of a static camera, and (2) a board sequence, where the object is fixed on a ChArUco board to provide accurate camera poses via marker detection. The goal of NVS-HO is to learn a NVS model that captures the full appearance of an object from (1), whereas (2) provides the ground-truth images used for evaluation. To establish baselines, we consider both a classical SfM pipeline and a state-of-the-art pre-trained feed-forward neural network (VGGT) as pose estimators, and train NVS models based on NeRF and Gaussian Splatting. Our experiments reveal significant performance gaps in current methods under unconstrained handheld conditions, highlighting the need for more robust approaches. NVS-HO thus offers a challenging real-world benchmark to drive progress in RGB-based novel view synthesis of handheld objects.

Problem

• Novel View Synthesis
• Handheld Objects
• RGB-based Reconstruction
• Real-world Benchmark

Innovation

• Novel View Synthesis
• Handheld Objects
• RGB-only Benchmark
• NeRF
• Gaussian Splatting