RacketVision: A Multiple Racket Sports Benchmark for Unified Ball and Racket Analysis

2025-11-21

AI Summary

This work addresses three core challenges in racket-sport analytics—fine-grained ball tracking, joint-level racket pose estimation, and ball trajectory prediction—across badminton, tennis, and table tennis. To this end, we introduce the first large-scale, multi-view video dataset covering all three sports, featuring fine-grained annotations of 3D ball center positions and multi-joint racket poses. We propose a unified multi-task benchmark and identify that naive concatenation of multimodal features degrades performance. Accordingly, we design a Cross-Attention mechanism to enable adaptive fusion of visual and geometric features. Experiments demonstrate that our approach significantly outperforms strong unimodal baselines on trajectory prediction, validating the efficacy of cross-modal collaborative modeling. Our dataset, benchmark, and method establish a new paradigm and reproducible foundation for multimodal learning in sports analytics.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

We introduce RacketVision, a novel dataset and benchmark for advancing computer vision in sports analytics, covering table tennis, tennis, and badminton. The dataset is the first to provide large-scale, fine-grained annotations for racket pose alongside traditional ball positions, enabling research into complex human-object interactions. It is designed to tackle three interconnected tasks: fine-grained ball tracking, articulated racket pose estimation, and predictive ball trajectory forecasting. Our evaluation of established baselines reveals a critical insight for multi-modal fusion: while naively concatenating racket pose features degrades performance, a CrossAttention mechanism is essential to unlock their value, leading to trajectory prediction results that surpass strong unimodal baselines. RacketVision provides a versatile resource and a strong starting point for future research in dynamic object tracking, conditional motion forecasting, and multimodal analysis in sports. Project page at https://github.com/OrcustD/RacketVision

Problem

• Unified ball and racket tracking across multiple sports disciplines
• Articulated racket pose estimation with fine-grained annotations
• Multimodal trajectory prediction through effective feature fusion

Innovation

• CrossAttention mechanism for multimodal fusion
• Fine-grained racket pose and ball tracking
• Unified dataset for multiple racket sports