Event-Driven Proactive Assistive Manipulation with Grounded Vision-Language Planning

2026-03-25

AI Summary

This work proposes an event-driven proactive assistance framework for collaborative robots, addressing the limitation of traditional systems that rely solely on explicit human instructions and lack the ability to infer user intent. By monitoring workspace state changes triggered by human-object interactions, the framework captures visual-language state snapshots upon event completion to infer task goals and determine whether intervention is warranted. When assistance is needed, it generates executable action sequences using constrained motion primitives and object references via integer IDs. This approach represents the first shift from request-driven to event-driven assistance, leveraging evidence from state transitions to enable anticipatory collaboration. Evaluated on a real-world desktop digit-block task, the method significantly improves the proactive completion rate in solvable scenarios while appropriately deferring action in unsolvable ones, demonstrating both effectiveness and robustness.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Assistance in collaborative manipulation is often initiated by user instructions, making high-level reasoning request-driven. In fluent human teamwork, however, partners often infer the next helpful step from the observed outcome of an action rather than waiting for instructions. Motivated by this, we introduce a shift from request-driven assistance to event-driven proactive assistance, where robot actions are initiated by workspace state transitions induced by human--object interactions rather than user-provided task instructions. To this end, we propose an event-driven framework that tracks interaction progress with an event monitor and, upon event completion, extracts stabilized pre/post snapshots that characterize the resulting state transition. Given the stabilized snapshots, the planner analyzes the implied state transition to infer a task-level goal and decide whether to intervene; if so, it generates a sequence of assistive actions. To make outputs executable and verifiable, we restrict actions to a set of action primitives and reference objects via integer IDs. We evaluate the framework on a real tabletop number-block collaboration task, demonstrating that explicit pre/post state-change evidence improves proactive completion on solvable scenes and appropriate waiting on unsolvable ones.

Problem

• event-driven assistance
• proactive manipulation
• human-robot collaboration
• state transition
• assistive robotics

Innovation

• event-driven assistance
• proactive manipulation
• vision-language planning
• state transition inference
• action primitives