CrowdMoGen: Zero-Shot Text-Driven Collective Motion Generation

2024-07-08

AI Summary

Existing text-to-motion generation methods struggle to scale to large-scale crowds and lack modeling capabilities for individual event responsiveness and spatial coherence—stemming from dual challenges in semantic scene planning (e.g., group organization, activity design, interaction coordination) and controllable, physically grounded motion synthesis. This paper proposes the first zero-shot collective motion generation framework: (1) an LLM-driven semantic scene planning module that integrates SMPL joint trajectory priors to enable crowd grouping and activity choreography; and (2) a joint-level activity-injected Transformer diffusion architecture supporting event-aligned, spatially coherent multi-agent motion sequence generation. Our method significantly outperforms state-of-the-art approaches in realism, event responsiveness, and spatial consistency. It establishes a new paradigm for urban simulation and interactive large-scale crowd modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Motion and Path Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While recent advances in text-to-motion generation have shown promising results, they typically assume all individuals are grouped as a single unit. Scaling these methods to handle larger crowds and ensuring that individuals respond appropriately to specific events remains a significant challenge. This is primarily due to the complexities of scene planning, which involves organizing groups, planning their activities, and coordinating interactions, and controllable motion generation. In this paper, we present CrowdMoGen, the first zero-shot framework for collective motion generation, which effectively groups individuals and generates event-aligned motion sequences from text prompts. 1) Being limited by the available datasets for training an effective scene planning module in a supervised manner, we instead propose a crowd scene planner that leverages pre-trained large language models (LLMs) to organize individuals into distinct groups. While LLMs offer high-level guidance for group divisions, they lack the low-level understanding of human motion. To address this, we further propose integrating an SMPL-based joint prior to generate context-appropriate activities, which consists of both joint trajectories and textual descriptions. 2) Secondly, to incorporate the assigned activities into the generative network, we introduce a collective motion generator that integrates the activities into a transformer-based network in a joint-wise manner, maintaining the spatial constraints during the multi-step denoising process. Extensive experiments demonstrate that CrowdMoGen significantly outperforms previous approaches, delivering realistic, event-driven motion sequences that are spatially coherent. As the first framework of collective motion generation, CrowdMoGen has the potential to advance applications in urban simulation, crowd planning, and other large-scale interactive environments.

Problem

• Generating realistic collective motion from text prompts
• Organizing individuals into groups for scene planning
• Ensuring event-aligned and spatially coherent motion sequences

Innovation

• Leverages LLMs for crowd scene planning
• Integrates SMPL-based joint prior for activities
• Uses transformer-based network for motion generation