Di-NeRF: Distributed NeRF for Collaborative Learning With Relative Pose Refinement

2024-02-02

IEEE Robotics and Automation Letters

AI Summary

Multi-robot collaborative 3D scene reconstruction under communication constraints and inaccurate initial pose estimates remains challenging. Method: This paper proposes the first fully decentralized NeRF co-training framework, wherein each robot trains its own NeRF solely from local visual observations, asynchronously exchanges model parameters over a mesh network, and jointly optimizes relative poses and radiance fields in an end-to-end, differentiable manner—embedding pose refinement directly into NeRF training. The approach requires no central coordinator and achieves robust reconstruction even with low-accuracy initial poses. Contribution/Results: Experiments demonstrate significant improvements in reconstruction completeness and geometric accuracy over single-agent baselines. On both real-world and synthetic datasets, the framework reduces communication overhead by over 40%, while maintaining scalability and mapping robustness under sparse connectivity and pose uncertainty.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Multiagent Systems: Coordination and Collaboration

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Collaborative mapping of unknown environments can be done faster and more robustly than a single robot. However, a collaborative approach requires a distributed paradigm to be scalable and deal with communication issues. This work presents a fully distributed algorithm enabling a group of robots to collectively optimize the parameters of a Neural Radiance Field (NeRF). The algorithm involves the communication of each robot's trained NeRF parameters over a mesh network, where each robot trains its NeRF and has access to its own visual data only. Additionally, the relative poses of all robots are jointly optimized alongside the model parameters, enabling mapping with less accurate relative camera poses. We show that multi-robot systems can benefit from differentiable and robust 3D reconstruction optimized from multiple NeRFs. Experiments on real-world and synthetic data demonstrate the efficiency of the proposed algorithm.

Problem

• Enables collaborative NeRF optimization among multiple robots
• Refines relative poses alongside NeRF parameters for accuracy
• Distributed approach enhances scalability and communication robustness

Innovation

• Distributed NeRF optimization among multiple robots
• Joint relative pose refinement alongside model parameters
• Mesh network communication for collaborative learning