Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation

2026-06-29

AI Summary

This work addresses the stability-plasticity dilemma faced by motion-language agents in dynamic environments when continuously learning new action concepts. The authors propose a bidirectional continual learning framework built upon a frozen large language model, integrating low-rank adaptation (LoRA) with an autoencoder-driven hard-routing mixture-of-experts (MoE) mechanism. This approach automatically selects task-specific experts without requiring task labels, effectively isolating knowledge to achieve near-zero forgetting. Evaluated on a five-task continual learning benchmark constructed from HumanML3D, the method maintains high performance in both text-to-motion (T2M) generation and motion-to-text (M2T) description tasks. Notably, hard expert selection significantly outperforms soft fusion strategies, underscoring the critical role of expert isolation in continual learning for multimodal motion-language understanding.

Abstract

Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M). While foundational models have achieved strong performance in static settings, autonomous agents operating in dynamic environments must continuously incorporate new motion concepts -- such as novel athletic styles or specialized gestures -- without catastrophic forgetting of previously acquired skills. We investigate the stability-plasticity trade-off in bidirectional motion-language learning under sequential task exposure. Building on a frozen large language model backbone, we introduce low-rank adaptation (LoRA) variants designed to mitigate inter-task interference. We specifically propose mixture-of-experts architectures that utilize an autoencoder-based router to select task-specific experts at inference time, so that no task-label is needed. To evaluate these methods, we establish a reproducible five-task benchmark derived from HumanML3D through semantic clustering of motion descriptions. Our experimental results demonstrate near-zero forgetting across both M2T and T2M directions while maintaining high generation and captioning quality. Furthermore, we show that hard expert selection via routing significantly outperforms soft expert blending in quality metrics, indicating that preserving expert isolation is critical for maintaining performance in our continual learning setting. Finally, we observe that a divergence between token-level accuracy and downstream generation quality may occur, highlighting the need for more comprehensive evaluation protocols in future research on lifelong motion-language agents.

Problem

• continual learning
• motion-language agents
• catastrophic forgetting
• text-to-motion
• motion-to-text

Innovation

• Continual Learning
• LoRA Variants
• Mixture-of-Experts
• Motion-Language Agents
• Task Routing