OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation

2025-06-23

AI Summary

Existing audio-driven human animation methods primarily focus on facial synthesis, struggling to achieve natural full-body motion, precise temporal synchronization, and fine-grained textual control. To address these limitations, we propose a novel framework for generating full-body virtual avatar videos. Our approach introduces a pixel-level multi-scale audio embedding mechanism to enhance spatiotemporal modeling of audio features within the diffusion model’s latent space. Coupled with LoRA-based fine-tuning, it enables high-fidelity lip synchronization and fluid whole-body motion while preserving strong text controllability. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches on facial and upper-body video generation: lip-sync error decreases by 21.3%, and motion fluency (measured by FID) improves by 34.7%. The framework supports diverse high-quality applications, including podcast avatars, human-computer interaction, and singing-and-dancing synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Significant progress has been made in audio-driven human animation, while most existing methods focus mainly on facial movements, limiting their ability to create full-body animations with natural synchronization and fluidity. They also struggle with precise prompt control for fine-grained generation. To tackle these challenges, we introduce OmniAvatar, an innovative audio-driven full-body video generation model that enhances human animation with improved lip-sync accuracy and natural movements. OmniAvatar introduces a pixel-wise multi-hierarchical audio embedding strategy to better capture audio features in the latent space, enhancing lip-syncing across diverse scenes. To preserve the capability for prompt-driven control of foundation models while effectively incorporating audio features, we employ a LoRA-based training approach. Extensive experiments show that OmniAvatar surpasses existing models in both facial and semi-body video generation, offering precise text-based control for creating videos in various domains, such as podcasts, human interactions, dynamic scenes, and singing. Our project page is https://omni-avatar.github.io/.

Problem

• Generating full-body animations with natural audio synchronization
• Enhancing lip-sync accuracy and movement fluidity in animations
• Achieving precise text-based control for fine-grained video generation

Innovation

• Adaptive full-body animation with natural synchronization
• Pixel-wise multi-hierarchical audio embedding strategy
• LoRA-based training for prompt-driven control