SM$^2$ITH: Safe Mobile Manipulation with Interactive Human Prediction via Task-Hierarchical Bilevel Model Predictive Control

2025-11-21

AI Summary

To address the safety-critical challenge of collaborative navigation and manipulation for mobile manipulators in dynamic human environments, this paper proposes a dual-layer Model Predictive Control (MPC) framework integrating task hierarchy with human–robot interaction prediction. Methodologically, we introduce, for the first time, a learning-based human motion prediction model into a Hierarchical Task MPC (HTMPC) architecture, establishing a closed-loop, co-optimized human–robot dynamic planning framework that enables priority-aware task scheduling and real-time safety coordination. Our key contributions are: (1) the seamless coupling of interactive human motion prediction with hierarchical task control; and (2) empirical validation on Stretch 3 and Ridgeback-UR10 platforms demonstrating significant improvements in both safety and task completion efficiency—compared to weighted-objective approaches and open-loop human models—across delivery, pick-and-place, and adversarial interaction tasks.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Human-Robot Interaction
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Mobile manipulators are designed to perform complex sequences of navigation and manipulation tasks in human-centered environments. While recent optimization-based methods such as Hierarchical Task Model Predictive Control (HTMPC) enable efficient multitask execution with strict task priorities, they have so far been applied mainly to static or structured scenarios. Extending these approaches to dynamic human-centered environments requires predictive models that capture how humans react to the actions of the robot. This work introduces Safe Mobile Manipulation with Interactive Human Prediction via Task-Hierarchical Bilevel Model Predictive Control (SM$^2$ITH), a unified framework that combines HTMPC with interactive human motion prediction through bilevel optimization that jointly accounts for robot and human dynamics. The framework is validated on two different mobile manipulators, the Stretch 3 and the Ridgeback-UR10, across three experimental settings: (i) delivery tasks with different navigation and manipulation priorities, (ii) sequential pick-and-place tasks with different human motion prediction models, and (iii) interactions involving adversarial human behavior. Our results highlight how interactive prediction enables safe and efficient coordination, outperforming baselines that rely on weighted objectives or open-loop human models.

Problem

• Extends hierarchical control to dynamic human environments
• Integrates interactive human motion prediction with robot planning
• Ensures safe mobile manipulation through bilevel optimization

Innovation

• Bilevel optimization integrates robot and human dynamics
• Hierarchical task MPC with interactive human motion prediction
• Unified framework validated on multiple mobile manipulators