Real-time High-fidelity Gaussian Human Avatars with Position-based Interpolation of Spatially Distributed MLPs

2025-04-17

AI Summary

To address the challenge of simultaneously preserving pose-dependent fine details and enabling real-time rendering in Gaussian-based human avatars for multi-view video, this paper proposes a dynamic Gaussian modeling framework that synergistically combines spatially distributed MLPs with position-weighted interpolation. Our key contributions are: (1) a novel spatially distributed MLP architecture that generates pose-adaptive Gaussian attributes via position-weighted interpolation; (2) a decoupled representation using Gaussian offset bases and learnable coefficients to explicitly model high-frequency geometric and appearance details; and (3) surface control point constraints to enhance generalization under novel poses. Integrating neural radiance field priors with 3D Gaussian splatting, our method achieves real-time rendering (>30 FPS). On standard benchmarks, it improves PSNR by 2.1 dB and reduces LPIPS by 18% for both novel-view and novel-pose synthesis, significantly outperforming existing state-of-the-art approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Learning with Manifolds

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Many works have succeeded in reconstructing Gaussian human avatars from multi-view videos. However, they either struggle to capture pose-dependent appearance details with a single MLP, or rely on a computationally intensive neural network to reconstruct high-fidelity appearance but with rendering performance degraded to non-real-time. We propose a novel Gaussian human avatar representation that can reconstruct high-fidelity pose-dependence appearance with details and meanwhile can be rendered in real time. Our Gaussian avatar is empowered by spatially distributed MLPs which are explicitly located on different positions on human body. The parameters stored in each Gaussian are obtained by interpolating from the outputs of its nearby MLPs based on their distances. To avoid undesired smooth Gaussian property changing during interpolation, for each Gaussian we define a set of Gaussian offset basis, and a linear combination of basis represents the Gaussian property offsets relative to the neutral properties. Then we propose to let the MLPs output a set of coefficients corresponding to the basis. In this way, although Gaussian coefficients are derived from interpolation and change smoothly, the Gaussian offset basis is learned freely without constraints. The smoothly varying coefficients combined with freely learned basis can still produce distinctly different Gaussian property offsets, allowing the ability to learn high-frequency spatial signals. We further use control points to constrain the Gaussians distributed on a surface layer rather than allowing them to be irregularly distributed inside the body, to help the human avatar generalize better when animated under novel poses. Compared to the state-of-the-art method, our method achieves better appearance quality with finer details while the rendering speed is significantly faster under novel views and novel poses.

Problem

• Real-time high-fidelity Gaussian human avatar rendering
• Capturing pose-dependent appearance details efficiently
• Improving rendering speed without sacrificing visual quality

Innovation

• Spatially distributed MLPs for detailed avatars
• Position-based interpolation of Gaussian properties
• Control points for better pose generalization