InsideOut: Integrated RGB-Radiative Gaussian Splatting for Comprehensive 3D Object Representation

2025-10-15

AI Summary

This work addresses the challenges of cross-modal 3D reconstruction from heterogeneous RGB (appearance details) and X-ray (internal structure) images—namely, severe modality discrepancy, difficult image registration, and scarcity of paired training data. We propose a radiography-aware Gaussian splatting method tailored for cross-modal reconstruction. Its core innovation lies in a hierarchical geometry-appearance joint optimization strategy, incorporating an X-ray projection consistency loss to enable, for the first time, end-to-end co-modeling of RGB and X-ray signals within the 3D Gaussian splatting framework. The method requires no strict pixel-level correspondence and supports non-invasive, unified 3D representation integrating both surface appearance and internal anatomy. Evaluated on a newly constructed paired dataset, our approach significantly improves surface visual fidelity and internal geometric consistency of reconstructions. It establishes a novel, interpretable, and high-fidelity multi-modal 3D modeling paradigm with direct applicability to medical diagnosis and non-destructive cultural heritage inspection.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We introduce InsideOut, an extension of 3D Gaussian splatting (3DGS) that bridges the gap between high-fidelity RGB surface details and subsurface X-ray structures. The fusion of RGB and X-ray imaging is invaluable in fields such as medical diagnostics, cultural heritage restoration, and manufacturing. We collect new paired RGB and X-ray data, perform hierarchical fitting to align RGB and X-ray radiative Gaussian splats, and propose an X-ray reference loss to ensure consistent internal structures. InsideOut effectively addresses the challenges posed by disparate data representations between the two modalities and limited paired datasets. This approach significantly extends the applicability of 3DGS, enhancing visualization, simulation, and non-destructive testing capabilities across various domains.

Problem

• Bridges RGB surface details with subsurface X-ray structures
• Addresses disparate data representations between imaging modalities
• Overcomes limited paired RGB and X-ray datasets availability

Innovation

• Integrates RGB and X-ray imaging via Gaussian splatting
• Uses hierarchical fitting to align multimodal Gaussian splats
• Proposes X-ray reference loss for consistent internal structures