UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields

2025-06-26

AI Summary

Existing NeRF-based segmentation methods rely solely on RGB data for semantic modeling, neglecting intrinsic material properties—limiting applicability in robotics manipulation, AR, and simulation, where precise material perception is essential. To address this, we introduce spectral unmixing into the NeRF framework for the first time, proposing Endmember-Driven Hyperspectral NeRF (HS-NeRF). HS-NeRF jointly achieves high-fidelity hyperspectral novel-view synthesis and unsupervised material segmentation: it represents material reflectance via a learnable endmember dictionary, models spatially continuous material abundance fields, and achieves material-level semantic disentanglement through endmember clustering. The framework further enables material-aware scene editing. Extensive experiments demonstrate that HS-NeRF significantly outperforms state-of-the-art methods in both spectral reconstruction accuracy and material segmentation metrics, validating its capability for high-fidelity material perception and controllable editing.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Neural Radiance Field (NeRF)-based segmentation methods focus on object semantics and rely solely on RGB data, lacking intrinsic material properties. This limitation restricts accurate material perception, which is crucial for robotics, augmented reality, simulation, and other applications. We introduce UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling joint hyperspectral novel view synthesis and unsupervised material segmentation. Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures, and per-point abundances capture their distribution. For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF enables scene editing by modifying learned endmember dictionaries for flexible material-based appearance manipulation. Extensive experiments validate our approach, demonstrating superior spectral reconstruction and material segmentation to existing methods. Project page: https://www.factral.co/UnMix-NeRF.

Problem

• Enables joint hyperspectral view synthesis and material segmentation
• Overcomes limitation of lacking intrinsic material properties in NeRF
• Allows unsupervised material clustering and scene appearance editing

Innovation

• Integrates spectral unmixing into NeRF
• Uses learned dictionary of global endmembers
• Enables unsupervised material clustering and editing