Multimodal Signal Processing For Thermo-Visible-Lidar Fusion In Real-time 3D Semantic Mapping

2026-01-14

AI Summary

Traditional SLAM systems struggle to simultaneously deliver high-fidelity geometric reconstruction and critical semantic information in complex environments, limiting robotic situational awareness in applications such as disaster assessment and industrial inspection. To address this, this work proposes a pixel-level multimodal approach that fuses visible and infrared imagery, projecting real-time LiDAR point clouds onto the fused image plane. High-temperature targets are segmented using thermal channels and embedded as a temperature-aware semantic layer within the 3D map. This method achieves, for the first time, pixel-aligned integration of thermal semantic information into real-time 3D semantic mapping, significantly enhancing immediate detection and labeling of high-temperature objects. The approach demonstrates practical utility in rapid disaster response and predictive industrial maintenance scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services

Abstract

In complex environments, autonomous robot navigation and environmental perception pose higher requirements for SLAM technology. This paper presents a novel method for semantically enhancing 3D point cloud maps with thermal information. By first performing pixel-level fusion of visible and infrared images, the system projects real-time LiDAR point clouds onto this fused image stream. It then segments heat source features in the thermal channel to instantly identify high temperature targets and applies this temperature information as a semantic layer on the final 3D map. This approach generates maps that not only have accurate geometry but also possess a critical semantic understanding of the environment, making it highly valuable for specific applications like rapid disaster assessment and industrial preventive maintenance.

Problem

• Multimodal Signal Processing
• Thermo-Visible-Lidar Fusion
• Real-time 3D Semantic Mapping
• SLAM
• Thermal Information

Innovation

• multimodal fusion
• thermal-visual-LiDAR integration
• real-time 3D semantic mapping
• heat source segmentation
• semantic SLAM