Learning Wheelchair Tennis Navigation from Broadcast Videos with Domain Knowledge Transfer and Diffusion Motion Planning

2024-09-29

arXiv.org

AI Summary

This work addresses wheelchair tennis navigation by proposing a zero-shot cross-domain knowledge transfer framework that directly distills expert motion policies from publicly available broadcast videos—without manual annotations or simulation-based pretraining. Methodologically, it integrates multi-view 3D reconstruction, diffusion model–driven motion planning in image space, domain-adaptive adversarial training, and 2D/3D co-optimized closed-loop control, forming a hierarchical position-based navigation architecture enabling real-time local trajectory planning for physical robots tracking actual ball trajectories. The core contribution is the first end-to-end “video-to-robot” policy transfer paradigm, circumventing conventional imitation learning’s reliance on sensor-specific modalities and action priors. Experiments demonstrate a 97.67% trajectory arrival success rate and a 68.49% real-time navigation success rate across full-scale courts.

Technology Category

• Intelligent Robots: Motion and Path Planning
• Computer Vision: Motion & Tracking
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we propose a novel and generalizable zero-shot knowledge transfer framework that distills expert sports navigation strategies from web videos into robotic systems with adversarial constraints and out-of-distribution image trajectories. Our pipeline enables diffusion-based imitation learning by reconstructing the full 3D task space from multiple partial views, warping it into 2D image space, closing the planning loop within this 2D space, and transfer constrained motion of interest back to task space. Additionally, we demonstrate that the learned policy can serve as a local planner in conjunction with position control. We apply this framework in the wheelchair tennis navigation problem to guide the wheelchair into the ball-hitting region. Our pipeline achieves a navigation success rate of 97.67% in reaching real-world recorded tennis ball trajectories with a physical robot wheelchair, and achieve a success rate of 68.49% in a real-world, real-time experiment on a full-sized tennis court.

Problem

• Transfer expert sports navigation from videos to robots
• Reconstruct 3D task space from partial 2D views
• Enable wheelchair tennis navigation for ball-hitting

Innovation

• Zero-shot knowledge transfer from web videos
• Diffusion-based imitation learning with 3D reconstruction
• Local planner integration with position control