RePerformer: Immersive Human-centric Volumetric Videos from Playback to Photoreal Reperformance

2025-03-15

AI Summary

Existing human-centric volumetric video methods are largely confined to dynamic scene replay or character animation, lacking high-fidelity reenactment capability for general dynamic scenes. To address this, we propose the first human-centric volumetric video framework enabling unified “replay → reenactment” modeling. Our approach introduces a hierarchical, disentangled Gaussian representation for motion and appearance, augmented by a semantic-aware alignment module and a deformation-transfer-based motion retargeting mechanism. Integrating Gaussian splatting, Morton encoding, a 2D position-to-attribute mapping CNN, and canonical-space modeling, our method achieves efficient multi-view reconstruction and photorealistic novel-pose rendering. Extensive evaluations on standard benchmarks demonstrate comprehensive superiority over state-of-the-art methods, establishing new paradigmatic benchmarks in reconstruction accuracy, reenactment fidelity, and generalization capability.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Human-centric volumetric videos offer immersive free-viewpoint experiences, yet existing methods focus either on replaying general dynamic scenes or animating human avatars, limiting their ability to re-perform general dynamic scenes. In this paper, we present RePerformer, a novel Gaussian-based representation that unifies playback and re-performance for high-fidelity human-centric volumetric videos. Specifically, we hierarchically disentangle the dynamic scenes into motion Gaussians and appearance Gaussians which are associated in the canonical space. We further employ a Morton-based parameterization to efficiently encode the appearance Gaussians into 2D position and attribute maps. For enhanced generalization, we adopt 2D CNNs to map position maps to attribute maps, which can be assembled into appearance Gaussians for high-fidelity rendering of the dynamic scenes. For re-performance, we develop a semantic-aware alignment module and apply deformation transfer on motion Gaussians, enabling photo-real rendering under novel motions. Extensive experiments validate the robustness and effectiveness of RePerformer, setting a new benchmark for playback-then-reperformance paradigm in human-centric volumetric videos.

Problem

• Unifies playback and re-performance in human-centric volumetric videos.
• Disentangles dynamic scenes into motion and appearance Gaussians.
• Enables photo-real rendering under novel motions using semantic-aware alignment.

Innovation

• Gaussian-based representation for volumetric videos
• Morton-based parameterization for efficient encoding
• Semantic-aware alignment for photo-real re-performance