Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment

2026-03-11

AI Summary

This work addresses catastrophic forgetting in imitation learning under limited memory and data settings for continuous task learning by proposing a lifelong imitation learning framework. The approach compactly stores and replays multimodal representations of visual observations, language instructions, and robot states in a shared latent space, while introducing an angular margin–based incremental feature adaptation mechanism to preserve discriminability across tasks. Evaluated on the LIBERO benchmark, the method achieves a new state of the art, improving the area under the curve (AUC) by 10–17 percentage points and reducing forgetting rates by up to 65%. Ablation studies confirm the effectiveness of each proposed component.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce a lifelong imitation learning framework that enables continual policy refinement across sequential tasks under realistic memory and data constraints. Our approach departs from conventional experience replay by operating entirely in a multimodal latent space, where compact representations of visual, linguistic, and robot's state information are stored and reused to support future learning. To further stabilize adaptation, we introduce an incremental feature adjustment mechanism that regularizes the evolution of task embeddings through an angular margin constraint, preserving inter-task distinctiveness. Our method establishes a new state of the art in the LIBERO benchmarks, achieving 10-17 point gains in AUC and up to 65% less forgetting compared to previous leading methods. Ablation studies confirm the effectiveness of each component, showing consistent gains over alternative strategies. The code is available at: https://github.com/yfqi/lifelong_mlr_ifa.

Problem

• lifelong imitation learning
• continual learning
• catastrophic forgetting
• multimodal representation
• task distinctiveness

Innovation

• lifelong imitation learning
• multimodal latent replay
• incremental feature adjustment
• angular margin constraint
• continual learning