CoachMe: Decoding Sport Elements with a Reference-Based Coaching Instruction Generation Model

2025-09-15

AI Summary

Existing motion analysis methods struggle to generate precise, interpretable sport-specific improvement suggestions from subtle kinematic discrepancies. This paper proposes a reference-driven generative model integrated with a multimodal motion understanding framework. It jointly models temporal dynamics and biomechanical constraints to explicitly learn fine-grained spatiotemporal-kinetic deviations between learner and reference motions, and emulates expert coaching reasoning to produce structured, actionable feedback. The model enables few-shot cross-sport transfer without requiring large-scale annotated data. Evaluated on figure skating and boxing tasks, it outperforms GPT-4o by 31.6% and 58.3% on the G-Eval metric, respectively. Generated instructions significantly improve error localization accuracy and the feasibility of corrective interventions. This work establishes a new paradigm for intelligent sports assistance—characterized by interpretability, domain adaptability, and cognitively grounded instruction generation.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Motion and Path Planning
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Motion instruction is a crucial task that helps athletes refine their technique by analyzing movements and providing corrective guidance. Although recent advances in multimodal models have improved motion understanding, generating precise and sport-specific instruction remains challenging due to the highly domain-specific nature of sports and the need for informative guidance. We propose CoachMe, a reference-based model that analyzes the differences between a learner's motion and a reference under temporal and physical aspects. This approach enables both domain-knowledge learning and the acquisition of a coach-like thinking process that identifies movement errors effectively and provides feedback to explain how to improve. In this paper, we illustrate how CoachMe adapts well to specific sports such as skating and boxing by learning from general movements and then leveraging limited data. Experiments show that CoachMe provides high-quality instructions instead of directions merely in the tone of a coach but without critical information. CoachMe outperforms GPT-4o by 31.6% in G-Eval on figure skating and by 58.3% on boxing. Analysis further confirms that it elaborates on errors and their corresponding improvement methods in the generated instructions. You can find CoachMe here: https://motionxperts.github.io/

Problem

• Generating precise sport-specific motion instructions for athletes
• Analyzing differences between learner and reference motions effectively
• Providing corrective feedback to improve athletic technique

Innovation

• Reference-based model analyzing motion differences
• Learning domain-knowledge and coach-like thinking
• Adapting to specific sports with limited data