Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations

2025-05-27

AI Summary

This work addresses the underutilization of suboptimal behavior signals in offline imitation learning (IL). We propose a novel contrastive demonstration framework leveraging both expert and suboptimal trajectories. Its core innovation is the first introduction of a differential KL divergence objective, which explicitly models discrepancies between state-action visitation distributions induced by expert versus suboptimal demonstrations—thereby incorporating suboptimal behavior as explicit negative signals into the learning objective. This objective remains convex when expert data dominates, enabling stable, non-adversarial end-to-end training. Our method integrates difference-of-convex (DC) optimization with joint state-action distribution modeling. Evaluated on standard offline IL benchmarks, it consistently outperforms state-of-the-art methods, achieving significant improvements in policy safety and generalization while effectively suppressing the reproduction of suboptimal behaviors.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes convex when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines.

Problem

• Learning from expert and undesirable demonstrations offline
• Optimizing difference of KL divergences for contrasting behaviors
• Avoiding adversarial training with a unified framework

Innovation

• Optimizes difference of KL divergences
• Convex objective with expert dominance
• Non-adversarial unified framework