MDMP: Multi-modal Diffusion for supervised Motion Predictions with uncertainty

2024-10-04

arXiv.org

AI Summary

Existing long-horizon human motion prediction methods rely solely on skeletal sequences or textual prompts, compromising accuracy, controllability, and uncertainty quantification. This paper proposes a skeleton-text dual-modal diffusion framework that jointly models spatiotemporal dynamics via a graph-structured Transformer—marking the first integration of such architecture for this task—and explicitly maps predictive uncertainty to joint-level confidence regions, enhancing spatial awareness in human-robot interaction. The method unifies multimodal diffusion, graph neural networks, cross-modal alignment, and uncertainty modeling. On multiple benchmarks, it reduces long-term prediction error (>1 second) by 18.7% and achieves strong calibration between estimated uncertainty and actual error (Spearman ρ = 0.92), significantly outperforming existing generative approaches. This advancement provides a robust foundation for safe, adaptive human-robot collaboration.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper introduces a Multi-modal Diffusion model for Motion Prediction (MDMP) that integrates and synchronizes skeletal data and textual descriptions of actions to generate refined long-term motion predictions with quantifiable uncertainty. Existing methods for motion forecasting or motion generation rely solely on either prior motions or text prompts, facing limitations with precision or control, particularly over extended durations. The multi-modal nature of our approach enhances the contextual understanding of human motion, while our graph-based transformer framework effectively capture both spatial and temporal motion dynamics. As a result, our model consistently outperforms existing generative techniques in accurately predicting long-term motions. Additionally, by leveraging diffusion models' ability to capture different modes of prediction, we estimate uncertainty, significantly improving spatial awareness in human-robot interactions by incorporating zones of presence with varying confidence levels for each body joint.

Problem

• Integrates skeletal data and text for motion prediction
• Improves long-term motion forecasting with uncertainty
• Enhances spatial awareness in human-robot interactions

Innovation

• Multi-modal Diffusion model integrates skeletal and textual data
• Graph-based transformer captures spatial-temporal motion dynamics
• Diffusion models estimate uncertainty for human-robot interactions