Veta-GS: View-dependent deformable 3D Gaussian Splatting for thermal infrared Novel-view Synthesis

2025-05-25

AI Summary

To address floating artifacts and edge blurring in thermal infrared novel-view synthesis—caused by transmission effects, emissivity variations, and low spatial resolution—this paper proposes a view-dependent deformable Gaussian field jointly optimized with a Thermal Feature Extractor (TFE). Within the 3D Gaussian Splatting framework, we introduce a camera-pose-driven deformation field for geometric adaptability and integrate a lightweight thermal feature encoder to model radiometric appearance. A MonoSSIM multi-scale perceptual loss is further designed to enhance structural fidelity. To our knowledge, this is the first method enabling joint geometry-appearance optimization specifically for the thermal infrared domain. Evaluated on the TI-NSD benchmark, our approach achieves a 12.6% PSNR improvement over prior methods, significantly suppressing floating artifacts and blur while supporting real-time rendering.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recently, 3D Gaussian Splatting (3D-GS) based on Thermal Infrared (TIR) imaging has gained attention in novel-view synthesis, showing real-time rendering. However, novel-view synthesis with thermal infrared images suffers from transmission effects, emissivity, and low resolution, leading to floaters and blur effects in rendered images. To address these problems, we introduce Veta-GS, which leverages a view-dependent deformation field and a Thermal Feature Extractor (TFE) to precisely capture subtle thermal variations and maintain robustness. Specifically, we design view-dependent deformation field that leverages camera position and viewing direction, which capture thermal variations. Furthermore, we introduce the Thermal Feature Extractor (TFE) and MonoSSIM loss, which consider appearance, edge, and frequency to maintain robustness. Extensive experiments on the TI-NSD benchmark show that our method achieves better performance over existing methods.

Problem

• Addresses floaters and blur in thermal infrared novel-view synthesis
• Captures subtle thermal variations using view-dependent deformation
• Improves robustness with Thermal Feature Extractor and MonoSSIM loss

Innovation

• View-dependent deformation field captures thermal variations
• Thermal Feature Extractor enhances robustness
• MonoSSIM loss considers appearance, edge, frequency