SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents

2025-06-05

AI Summary

Existing diffusion-based methods for 3D portrait generation struggle to simultaneously ensure identity fidelity, controllable body morphology, and animation readiness. SmartAvatar addresses this by introducing an autonomous, iterative framework powered by a vision-language model (VLM) agent, capable of generating fully rigged, animatable 3D human avatars from a single image or text prompt. Its core innovation is a closed-loop verification mechanism: a VLM agent jointly evaluates facial similarity, anatomical plausibility, and prompt alignment, then issues natural-language instructions to guide parameter refinement. The method integrates parametric human models (e.g., SMPL-X), neural rendering, and large language model–driven procedural generation. Experiments demonstrate that SmartAvatar outperforms state-of-the-art methods across mesh quality, identity preservation, attribute accuracy, and animation usability. Moreover, it enables real-time customization on consumer-grade hardware.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Embodied AI

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search

Abstract

SmartAvatar is a vision-language-agent-driven framework for generating fully rigged, animation-ready 3D human avatars from a single photo or textual prompt. While diffusion-based methods have made progress in general 3D object generation, they continue to struggle with precise control over human identity, body shape, and animation readiness. In contrast, SmartAvatar leverages the commonsense reasoning capabilities of large vision-language models (VLMs) in combination with off-the-shelf parametric human generators to deliver high-quality, customizable avatars. A key innovation is an autonomous verification loop, where the agent renders draft avatars, evaluates facial similarity, anatomical plausibility, and prompt alignment, and iteratively adjusts generation parameters for convergence. This interactive, AI-guided refinement process promotes fine-grained control over both facial and body features, enabling users to iteratively refine their avatars via natural-language conversations. Unlike diffusion models that rely on static pre-trained datasets and offer limited flexibility, SmartAvatar brings users into the modeling loop and ensures continuous improvement through an LLM-driven procedural generation and verification system. The generated avatars are fully rigged and support pose manipulation with consistent identity and appearance, making them suitable for downstream animation and interactive applications. Quantitative benchmarks and user studies demonstrate that SmartAvatar outperforms recent text- and image-driven avatar generation systems in terms of reconstructed mesh quality, identity fidelity, attribute accuracy, and animation readiness, making it a versatile tool for realistic, customizable avatar creation on consumer-grade hardware.

Problem

• Generates 3D human avatars from text or single photo
• Ensures precise control over identity and body shape
• Provides animation-ready avatars with customizable features

Innovation

• VLM AI agents guide avatar generation process
• Autonomous verification loop ensures quality refinement
• LLM-driven procedural generation enables customization