Show, Don't Tell: Detecting Novel Objects by Watching Human Videos

2026-03-13

AI Summary

Existing robotic systems struggle to rapidly recognize out-of-distribution novel objects from human demonstrations: closed-set detectors fail, while open-set approaches rely on cumbersome language prompts. This work proposes a “show-don’t-tell” paradigm that leverages self-supervised learning to automatically construct training data and generate supervision signals directly from human manipulation videos, enabling the customization of novel object detectors without any textual descriptions. The method achieves end-to-end rapid adaptation of object detectors and significantly outperforms current state-of-the-art techniques on real robotic platforms, substantially improving task completion rates.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

How can a robot quickly identify and recognize new objects shown to it during a human demonstration? Existing closed-set object detectors frequently fail at this because the objects are out-of-distribution. While open-set detectors (e.g., VLMs) sometimes succeed, they often require expensive and tedious human-in-the-loop prompt engineering to uniquely recognize novel object instances. In this paper, we present a self-supervised system that eliminates the need for tedious language descriptions and expensive prompt engineering by training a bespoke object detector on an automatically created dataset, supervised by the human demonstration itself. In our approach, "Show, Don't Tell," we show the detector the specific objects of interest during the demonstration, rather than telling the detector about these objects via complex language descriptions. By bypassing language altogether, this paradigm enables us to quickly train bespoke detectors tailored to the relevant objects observed in human task demonstrations. We develop an integrated on-robot system to deploy our "Show, Don't Tell" paradigm of automatic dataset creation and novel object-detection on a real-world robot. Empirical results demonstrate that our pipeline significantly outperforms state-of-the-art detection and recognition methods for manipulated objects, leading to improved task completion for the robot.

Problem

• novel object detection
• human demonstration
• open-set detection
• robot perception
• out-of-distribution objects

Innovation

• self-supervised learning
• novel object detection
• human demonstration
• vision-based robotics
• prompt-free detection