Multi-View Pose-Agnostic Change Localization with Zero Labels

2024-12-05

arXiv.org

AI Summary

To address the challenge of environmental change perception by autonomous agents under unconstrained, unlabeled, and pose-unknown conditions, this paper introduces the first zero-label, pose-agnostic multi-view change localization method. Given only five post-change images, our approach constructs a change-aware 3D Gaussian Splatting (3DGS) scene representation, enabling unseen-view change mask generation via multi-view geometric consistency modeling, self-supervised change-channel learning, and a rendering-mask decoupling mechanism. Our contributions are threefold: (1) the first joint change modeling framework that requires neither pose priors nor manual annotations; (2) the first real-world multi-view change detection benchmark incorporating illumination variations; and (3) state-of-the-art performance on complex multi-object scenes, achieving 1.7× and 1.5× improvements in mIoU and F1-score, respectively, over prior methods.

Technology Category

• Computer Vision: Scene Analysis & Understanding
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Autonomous agents often require accurate methods for detecting and localizing changes in their environment, particularly when observations are captured from unconstrained and inconsistent viewpoints. We propose a novel label-free, pose-agnostic change detection method that integrates information from multiple viewpoints to construct a change-aware 3D Gaussian Splatting (3DGS) representation of the scene. With as few as 5 images of the post-change scene, our approach can learn an additional change channel in a 3DGS and produce change masks that outperform single-view techniques. Our change-aware 3D scene representation additionally enables the generation of accurate change masks for unseen viewpoints. Experimental results demonstrate state-of-the-art performance in complex multi-object scenes, achieving a 1.7x and 1.5x improvement in Mean Intersection Over Union and F1 score respectively over other baselines. We also contribute a new real-world dataset to benchmark change detection in diverse challenging scenes in the presence of lighting variations.

Problem

• Detect and localize environmental changes from inconsistent viewpoints.
• Develop a label-free, pose-agnostic change detection method.
• Improve change detection accuracy in complex multi-object scenes.

Innovation

• Label-free, pose-gnostic change detection method
• Multi-view 3D Gaussian Splatting representation
• Accurate change masks for unseen viewpoints