ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos

2026-04-12

AI Summary

Existing motion editing approaches rely on paired data or manual guidance, limiting their applicability to skill enhancement scenarios. This work proposes the first skill-aware motion editing framework that trains solely on unpaired expert videos. By leveraging masked language modeling to learn expert motion priors, the method automatically identifies and masks critical segments of novice motions during inference and maps them onto the expert motion manifold, enabling localized skill enhancement without supervision or human intervention. Evaluated across three distinct activities and eight technical tasks on the Ego-Exo4D and Kyokushin karate datasets, the approach outperforms current state-of-the-art supervised methods on multiple metrics.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Manipulation
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Visual feedback is critical for motor skill acquisition in sports and rehabilitation, and psychological studies show that observing near-perfect versions of one's own performance accelerates learning more effectively than watching expert demonstrations alone. We propose to enable such personalized feedback by automatically editing a person's motion to reflect higher skill. Existing motion editing approaches are poorly suited for this setting because they assume paired input-output data -- rare and expensive to curate for skill-driven tasks -- and explicit edit guidance at inference. We introduce ExpertEdit, a framework for skill-driven motion editing trained exclusively on unpaired expert video demonstrations. ExpertEdit learns an expert motion prior with a masked language modeling objective that infills masked motion spans with expert-level refinements. At inference, novice motion is masked at skill-critical moments and projected into the learned expert manifold, producing localized skill improvements without paired supervision or manual edit guidance. Across eight diverse techniques and three sports from Ego-Exo4D and Karate Kyokushin, ExpertEdit outperforms state-of-the-art supervised motion editing methods on multiple metrics of motion realism and expert quality. Project page: https://vision.cs.utexas.edu/projects/expert_edit/ .

Problem

• motion editing
• skill acquisition
• unpaired data
• expert demonstration
• personalized feedback

Innovation

• skill-aware motion editing
• unpaired expert videos
• masked language modeling
• expert motion prior
• motion manifold projection