CalTennis: Large Multi-View Tennis Video Dataset and Benchmark of Monocular-to-3D Pose Estimation

2026-06-18

AI Summary

Existing monocular 3D human pose estimation methods lack effective evaluation in real-world sports scenarios, particularly exhibiting blind spots in depth estimation, foot-ground contact detection, and motion stability. To address this gap, this work introduces CalTennis, a large-scale, multi-view synchronized tennis video dataset comprising over 11 million frames from 40 players, along with a standardized data collection pipeline that requires no specialized equipment and an automated camera calibration procedure. The study establishes the first multi-view benchmark for expert-level athletic motions and proposes a novel label-free, geometry-based 3D evaluation protocol derived from multi-view constraints. Additionally, it introduces two new evaluation metrics focusing on foot dynamics and temporal stability. Experiments reveal that while state-of-the-art models accurately recover joint angles, they struggle significantly with depth and contact estimation; the proposed metrics effectively expose these failure modes, offering clear guidance for future algorithmic improvements.

Abstract

The Caltech Tennis Dataset (CalTennis) is a large-scale video benchmark for evaluating monocular-to-3D pose estimation in the wild. CalTennis comprises over 11 million frames (51 hours) of tennis practice and match play from 40 players, captured with 2-6 synchronized cameras at 60 Hz. It is 10 times larger than existing in-the-wild human motion video datasets and 3 times larger than existing MOCAP-ground-truthed datasets, and it is the first large-scale benchmark to provide synchronized multi-view recordings of expert athletic motion. The multi-view setup enables inexpensive, label-free evaluation of monocular-to-3D pose estimation algorithms. We describe a simple, standardized protocol that enables data collection without specialized equipment or expertise, along with fully automated video calibration and synchronization. Benchmarking state-of-the-art monocular-to-3D pose methods on CalTennis, we find that while 3D joint angle recovery is now quite accurate, all models struggle to estimate depth and foot contact consistently. We further propose two novel performance metrics, footwork and stability, as well as qualitatively study body shape inconsistency. These metrics expose previously underexplored failure modes and point to concrete opportunities for improvement in pose estimation and action analysis.

Problem

• monocular-to-3D pose estimation
• depth estimation
• foot contact
• in-the-wild human motion
• pose estimation benchmark

Innovation

• multi-view video dataset
• monocular-to-3D pose estimation
• automatic calibration and synchronization
• footwork metric
• stability metric