TubeDAgger: Reducing the Number of Expert Interventions with Stochastic Reach-Tubes

2025-10-01

AI Summary

To address the high frequency of expert interventions and reliance on manually tuned thresholds in interactive imitation learning, this paper proposes a risk-aware intervention mechanism based on Stochastic Reachable Tubes (SRT). Methodologically, it is the first to adapt SRT—a technique from dynamical systems verification—to imitation learning, modeling the probabilistic state evolution during policy execution to quantify real-time loss-of-control risk and trigger adaptive expert takeover—eliminating environment-specific thresholds and replacing conventional confidence-based classification. Contributions include: (1) theoretically verifiable and data-agnostic expert intervention decisions; (2) support for online policy retraining and enhanced robustness; and (3) significant reduction in expert interventions across multiple simulation environments (average 42.7% decrease), while maintaining safety guarantees and policy performance stability.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Interactive Imitation Learning deals with training a novice policy from expert demonstrations in an online fashion. The established DAgger algorithm trains a robust novice policy by alternating between interacting with the environment and retraining of the network. Many variants thereof exist, that differ in the method of discerning whether to allow the novice to act or return control to the expert. We propose the use of stochastic reachtubes - common in verification of dynamical systems - as a novel method for estimating the necessity of expert intervention. Our approach does not require fine-tuning of decision thresholds per environment and effectively reduces the number of expert interventions, especially when compared with related approaches that make use of a doubt classification model.

Problem

• Reducing expert interventions in interactive imitation learning systems
• Estimating intervention need using stochastic reach-tube verification methods
• Eliminating environment-specific threshold tuning for intervention decisions

Innovation

• Uses stochastic reachtubes for intervention estimation
• Eliminates environment-specific threshold fine-tuning
• Reduces expert interventions compared to doubt models