Distilling Realizable Students from Unrealizable Teachers

2025-05-14

AI Summary

This paper addresses policy distillation under privileged information: the teacher observes the full state, whereas the student accesses only partial observations—inducing information asymmetry, distributional shift, and policy degradation. Existing approaches either degrade teacher capability to generate realizable demonstrations or compel the student to blindly explore unobserved states, both yielding low sample efficiency. We propose an active querying–correction mechanism and intelligent reinitialization to construct recoverable trajectories within the student’s observable subspace, avoiding forced imitation of unrealizable teacher policies. Our method integrates adaptive-query imitation learning with recovery-state–based reinforcement learning, requiring no teacher modification or auxiliary exploration. Evaluated on both simulation and real-robot tasks, our approach significantly improves training efficiency and final performance, consistently outperforming standard teacher–student distillation baselines.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We study policy distillation under privileged information, where a student policy with only partial observations must learn from a teacher with full-state access. A key challenge is information asymmetry: the student cannot directly access the teacher's state space, leading to distributional shifts and policy degradation. Existing approaches either modify the teacher to produce realizable but sub-optimal demonstrations or rely on the student to explore missing information independently, both of which are inefficient. Our key insight is that the student should strategically interact with the teacher --querying only when necessary and resetting from recovery states --to stay on a recoverable path within its own observation space. We introduce two methods: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration. We validate our methods in both simulated and real-world robotic tasks, demonstrating significant improvements over standard teacher-student baselines in training efficiency and final performance. The project website is available at : https://portal-cornell.github.io/CritiQ_ReTRy/

Problem

• Student learns from teacher with partial observations
• Information asymmetry causes policy degradation
• Efficient interaction between student and teacher needed

Innovation

• Student queries teacher adaptively for corrections
• Reinforcement learning initializes efficient exploration
• Combines imitation and reinforcement learning methods