Surface-SOS: Self-Supervised Object Segmentation via Neural Surface Representation

2024-03-12

IEEE Transactions on Image Processing

AI Summary

This paper addresses unsupervised object surface recognition and segmentation from unlabeled multi-view images. We propose a self-supervised neural surface representation method. Our approach introduces, for the first time, NeRF-enhanced signed distance function (SDF) modeling into self-supervised segmentation, employing a dual-module scene representation that explicitly decouples geometry from appearance. Training is fully unsupervised, leveraging multi-view geometric and texture consistency constraints. Additionally, we design a coarse-mask-guided single-view refinement mechanism to improve segmentation precision. Crucially, our method requires no static background assumptions, temporal video sequences, or human annotations. Extensive experiments on benchmarks—including LLFF and CO3D—as well as real-world scenes demonstrate substantial improvements over NeRF-based baselines and supervised single-view methods, yielding finer-grained, more accurate, and robust segmentation masks.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Self-supervised Object Segmentation (SOS) aims to segment objects without any annotations. Under conditions of multi-camera inputs, the structural, textural and geometrical consistency among each view can be leveraged to achieve fine-grained object segmentation. To make better use of the above information, we propose Surface representation based Self-supervised Object Segmentation (Surface-SOS), a new framework to segment objects for each view by 3D surface representation from multi-view images of a scene. To model high-quality geometry surfaces for complex scenes, we design a novel scene representation scheme, which decomposes the scene into two complementary neural representation modules respectively with a Signed Distance Function (SDF). Moreover, Surface-SOS is able to refine single-view segmentation with multi-view unlabeled images, by introducing coarse segmentation masks as additional input. To the best of our knowledge, Surface-SOS is the first self-supervised approach that leverages neural surface representation to break the dependence on large amounts of annotated data and strong constraints. These constraints typically involve observing target objects against a static background or relying on temporal supervision in videos. Extensive experiments on standard benchmarks including LLFF, CO3D, BlendedMVS, TUM and several real-world scenes show that Surface-SOS always yields finer object masks than its NeRF-based counterparts and surpasses supervised single-view baselines remarkably. Code is available at: https://github.com/zhengxyun/Surface-SOS.

Problem

• Multi-view Image Segmentation
• Unsupervised Learning
• Consistency Constraint

Innovation

• Self-supervised Learning
• Neural Surface Representation
• Multi-view Imagery