Zero-shot Interactive Perception

2026-02-20

AI Summary

In partially observable environments with complex occlusions, robots struggle to accurately interpret semantic scene content and execute tasks using passive perception alone. To address this challenge, this work proposes Zero-Shot Interactive Perception (ZS-IP), a framework that enhances perception through active, multi-strategy interactions such as pushing and grasping. ZS-IP introduces a novel 2D visual representation termed “pushlines” and integrates a memory-augmented vision-language model to enable semantic query responses without task-specific training. The system combines an augmented observation module—incorporating keypoints and pushlines—a memory-guided action policy, and a seven-degree-of-freedom robotic arm controller. Evaluated on the Franka Panda platform, ZS-IP significantly outperforms passive approaches like MOKA in pushing tasks while effectively preserving the integrity of non-target objects.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Visual Reasoning & Symbolic Representations
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Interactive perception (IP) enables robots to extract hidden information in their workspace and execute manipulation plans by physically interacting with objects and altering the state of the environment -- crucial for resolving occlusions and ambiguity in complex, partially observable scenarios. We present Zero-Shot IP (ZS-IP), a novel framework that couples multi-strategy manipulation (pushing and grasping) with a memory-driven Vision Language Model (VLM) to guide robotic interactions and resolve semantic queries. ZS-IP integrates three key components: (1) an Enhanced Observation (EO) module that augments the VLM's visual perception with both conventional keypoints and our proposed pushlines -- a novel 2D visual augmentation tailored to pushing actions, (2) a memory-guided action module that reinforces semantic reasoning through context lookup, and (3) a robotic controller that executes pushing, pulling, or grasping based on VLM output. Unlike grid-based augmentations optimized for pick-and-place, pushlines capture affordances for contact-rich actions, substantially improving pushing performance. We evaluate ZS-IP on a 7-DOF Franka Panda arm across diverse scenes with varying occlusions and task complexities. Our experiments demonstrate that ZS-IP outperforms passive and viewpoint-based perception techniques such as Mark-Based Visual Prompting (MOKA), particularly in pushing tasks, while preserving the integrity of non-target elements.

Problem

• Interactive Perception
• Occlusion Handling
• Semantic Ambiguity
• Robotic Manipulation
• Partial Observability

Innovation

• Zero-shot Interactive Perception
• pushlines
• Vision Language Model
• memory-guided action
• affordance-aware manipulation