Rigidity-Aware 3D Gaussian Deformation from a Single Image

2025-09-26

AI Summary

Single-image 3D non-rigid deformation reconstruction remains a fundamental challenge in computer vision and graphics, with most existing approaches relying on multi-view videos and thus suffering from limited generalizability. This paper introduces DeformSplat, the first framework enabling high-fidelity 3D Gaussian deformation reconstruction from a single image. Our method integrates 3D Gaussian representations, differentiable rendering, and sparse visual correspondence matching. Key contributions include: (1) a differentiable Gaussian-to-pixel matching mechanism that bridges the domain gap between 3D Gaussian primitives and 2D observations; and (2) a two-stage rigid-region segmentation strategy—combining attention-guided initialization with refinement—to ensure geometric consistency throughout deformation. Experiments demonstrate that DeformSplat significantly outperforms prior single-image methods in both deformation accuracy and cross-instance generalization. Moreover, it successfully extends to downstream applications including frame interpolation and interactive editing.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Reconstructing object deformation from a single image remains a significant challenge in computer vision and graphics. Existing methods typically rely on multi-view video to recover deformation, limiting their applicability under constrained scenarios. To address this, we propose DeformSplat, a novel framework that effectively guides 3D Gaussian deformation from only a single image. Our method introduces two main technical contributions. First, we present Gaussian-to-Pixel Matching which bridges the domain gap between 3D Gaussian representations and 2D pixel observations. This enables robust deformation guidance from sparse visual cues. Second, we propose Rigid Part Segmentation consisting of initialization and refinement. This segmentation explicitly identifies rigid regions, crucial for maintaining geometric coherence during deformation. By combining these two techniques, our approach can reconstruct consistent deformations from a single image. Extensive experiments demonstrate that our approach significantly outperforms existing methods and naturally extends to various applications,such as frame interpolation and interactive object manipulation.

Problem

• Reconstructing object deformation from single image
• Bridging 3D Gaussian representations with 2D pixels
• Identifying rigid regions to maintain geometric coherence

Innovation

• 3D Gaussian deformation guided from single image
• Bridging 3D Gaussians and 2D pixels via matching
• Identifying rigid regions through segmentation refinement