Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models

2025-06-17

AI Summary

Real-world assisted teleoperation faces significant challenges in real-time human intent understanding due to the diversity of user intentions and the open-ended nature of operational environments; existing approaches are constrained by predefined scenarios or task-specific data distributions. This paper introduces the first teleoperation framework integrating the commonsense reasoning capabilities of vision-language models (VLMs), featuring an open-world perception module and a commonsense-driven intent inference mechanism that enables open-set perception, long-horizon task generalization, and zero-shot intent understanding. Additionally, we design an extensible skill library to support flexible mobile manipulation. Experimental results demonstrate that our method substantially improves task completion rates, reduces user cognitive load, and achieves higher user satisfaction compared to both direct teleoperation and state-of-the-art assisted baselines.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Assistive teleoperation, where control is shared between a human and a robot, enables efficient and intuitive human-robot collaboration in diverse and unstructured environments. A central challenge in real-world assistive teleoperation is for the robot to infer a wide range of human intentions from user control inputs and to assist users with correct actions. Existing methods are either confined to simple, predefined scenarios or restricted to task-specific data distributions at training, limiting their support for real-world assistance. We introduce Casper, an assistive teleoperation system that leverages commonsense knowledge embedded in pre-trained visual language models (VLMs) for real-time intent inference and flexible skill execution. Casper incorporates an open-world perception module for a generalized understanding of novel objects and scenes, a VLM-powered intent inference mechanism that leverages commonsense reasoning to interpret snippets of teleoperated user input, and a skill library that expands the scope of prior assistive teleoperation systems to support diverse, long-horizon mobile manipulation tasks. Extensive empirical evaluation, including human studies and system ablations, demonstrates that Casper improves task performance, reduces human cognitive load, and achieves higher user satisfaction than direct teleoperation and assistive teleoperation baselines.

Problem

• Infer diverse human intentions from teleoperation inputs
• Generalize assistance beyond predefined scenarios and tasks
• Enable flexible skill execution in unstructured environments

Innovation

• Uses vision language models for intent inference
• Open-world perception for novel scenes
• Skill library for diverse manipulation tasks