SLAM&Render: A Benchmark for the Intersection Between Neural Rendering, Gaussian Splatting and SLAM

2025-04-18

AI Summary

Existing benchmarks at the intersection of SLAM and neural rendering lack comprehensive support for temporal modeling, multimodal perception, and cross-view/lighting generalization. Method: We propose SLAM&Render—the first standardized benchmark for this interdisciplinary domain—comprising 40 synchronized multimodal sequences (RGB, depth, IMU, robot kinematics, and ground-truth poses) across five scene categories, four lighting conditions, and object rearrangements. It uniquely integrates SLAM’s temporal robustness and multi-sensor constraints with neural rendering’s viewpoint and illumination generalization requirements, and introduces robot kinematics data for the first time to enable robotic-arm SLAM evaluation. High-precision motion capture, structured scenes, and perturbed trajectories ensure strict train/test separation. Results: Experiments expose critical performance bottlenecks of NeRF, Gaussian Splatting, and related methods on coupled SLAM-rendering tasks, establishing a reproducible, extensible framework for joint evaluation.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Models and methods originally developed for novel view synthesis and scene rendering, such as Neural Radiance Fields (NeRF) and Gaussian Splatting, are increasingly being adopted as representations in Simultaneous Localization and Mapping (SLAM). However, existing datasets fail to include the specific challenges of both fields, such as multimodality and sequentiality in SLAM or generalization across viewpoints and illumination conditions in neural rendering. To bridge this gap, we introduce SLAM&Render, a novel dataset designed to benchmark methods in the intersection between SLAM and novel view rendering. It consists of 40 sequences with synchronized RGB, depth, IMU, robot kinematic data, and ground-truth pose streams. By releasing robot kinematic data, the dataset also enables the assessment of novel SLAM strategies when applied to robot manipulators. The dataset sequences span five different setups featuring consumer and industrial objects under four different lighting conditions, with separate training and test trajectories per scene, as well as object rearrangements. Our experimental results, obtained with several baselines from the literature, validate SLAM&Render as a relevant benchmark for this emerging research area.

Problem

• Benchmarking SLAM and neural rendering integration challenges
• Addressing multimodality and sequentiality in SLAM datasets
• Evaluating generalization across viewpoints and lighting conditions

Innovation

• Combines SLAM with neural rendering techniques
• Introduces multimodal synchronized dataset sequences
• Enables robot manipulator SLAM strategy assessment