Feasibility-aware Imitation Learning from Observation with Multimodal Feedback

2026-02-16

AI Summary

This work addresses the challenge that human demonstrations often yield infeasible robot actions due to kinematic and dynamic discrepancies between humans and robots, compounded by the absence of robot-action labels. To this end, the authors propose FABCO, a novel approach that integrates robot dynamics–based feasibility assessment with multimodal feedback—visual and tactile—into observation-based imitation learning. This framework provides real-time guidance to demonstrators, encouraging the generation of more feasible actions, while simultaneously suppressing the influence of infeasible trajectories during policy training. Experimental results across two tasks involving 15 human participants demonstrate that FABCO improves imitation learning performance by over 3.2× compared to a baseline without feasibility feedback, effectively enhancing demonstration quality and enabling robust policy learning.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Imitation learning frameworks that learn robot control policies from demonstrators' motions via hand-mounted demonstration interfaces have attracted increasing attention. However, due to differences in physical characteristics between demonstrators and robots, this approach faces two limitations: i) the demonstration data do not include robot actions, and ii) the demonstrated motions may be infeasible for robots. These limitations make policy learning difficult. To address them, we propose Feasibility-Aware Behavior Cloning from Observation (FABCO). FABCO integrates behavior cloning from observation, which complements robot actions using robot dynamics models, with feasibility estimation. In feasibility estimation, the demonstrated motions are evaluated using a robot-dynamics model, learned from the robot's execution data, to assess reproducibility under the robot's dynamics. The estimated feasibility is used for multimodal feedback and feasibility-aware policy learning to improve the demonstrator's motions and learn robust policies. Multimodal feedback provides feasibility through the demonstrator's visual and haptic senses to promote feasible demonstrated motions. Feasibility-aware policy learning reduces the influence of demonstrated motions that are infeasible for robots, enabling the learning of policies that robots can execute stably. We conducted experiments with 15 participants on two tasks and confirmed that FABCO improves imitation learning performance by more than 3.2 times compared to the case without feasibility feedback.

Problem

• Imitation Learning
• Feasibility
• Robot Control
• Demonstration
• Multimodal Feedback

Innovation

• Feasibility-aware imitation learning
• Behavior cloning from observation
• Multimodal feedback
• Robot dynamics modeling
• Policy learning