CLEVER: Stream-based Active Learning for Robust Semantic Perception from Human Instructions

2025-07-21

AI Summary

To address the semantic perception failures and insufficient robustness of deep neural networks (DNNs) under streaming data conditions, this paper proposes a streaming active learning framework tailored for real-world robots. The method integrates domain priors via Bayesian modeling to detect perception failures online and actively solicit human feedback; it further employs a lightweight online update mechanism to incorporate human instructions into model optimization in real time, enabling continuous human-robot collaborative learning. This work presents the first end-to-end deployment of streaming active learning on humanoid robots performing deformable object manipulation tasks. Experiments demonstrate a significant improvement in DNN perceptual robustness within dynamic environments—reducing error rates by 32.7%—and user studies confirm its interaction efficiency and practical viability. Key contributions include: (1) the first robot-edge implementation of streaming active learning; (2) a Bayesian-driven failure detection and feedback integration mechanism; and (3) an online learning architecture supporting incremental human-robot co-adaptation.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Active Learning
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We propose CLEVER, an active learning system for robust semantic perception with Deep Neural Networks (DNNs). For data arriving in streams, our system seeks human support when encountering failures and adapts DNNs online based on human instructions. In this way, CLEVER can eventually accomplish the given semantic perception tasks. Our main contribution is the design of a system that meets several desiderata of realizing the aforementioned capabilities. The key enabler herein is our Bayesian formulation that encodes domain knowledge through priors. Empirically, we not only motivate CLEVER's design but further demonstrate its capabilities with a user validation study as well as experiments on humanoid and deformable objects. To our knowledge, we are the first to realize stream-based active learning on a real robot, providing evidence that the robustness of the DNN-based semantic perception can be improved in practice. The project website can be accessed at https://sites.google.com/view/thecleversystem.

Problem

• Enhance DNN-based semantic perception robustness via active learning
• Adapt DNNs online using human instructions for streaming data
• Implement stream-based active learning on real robots

Innovation

• Stream-based active learning for DNNs
• Online DNN adaptation via human instructions
• Bayesian formulation encoding domain knowledge