Smartphone Exergames with Real-Time Markerless Motion Capture: Challenges and Trade-offs

2025-07-09

AI Summary

Unmarked motion capture for mobile health and rehabilitation faces inherent trade-offs between accuracy and latency, alongside insufficient adaptability to user interaction. Method: This study proposes a lightweight, real-time exergame framework tailored for smartphones, integrating mobile-optimized AI pose estimation, low-latency video streaming, and adaptive gamified interaction design to achieve sub-second motion tracking on standard Android/iOS devices. Contributions/Results: (1) First systematic validation of a camera-only smartphone solution for home-based rehabilitation, demonstrating both technical feasibility and high user acceptability; (2) A co-optimization strategy for accuracy and real-time performance, reducing inference latency by 32% over baseline models while maintaining >92% keypoint detection accuracy; (3) A scalable, hardware-free remote rehabilitation prototype system that significantly improves user engagement and accessibility.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Markerless Motion Capture (MoCap) using smartphone cameras is a promising approach to making exergames more accessible and cost-effective for health and rehabilitation. Unlike traditional systems requiring specialized hardware, recent advancements in AI-powered pose estimation enable movement tracking using only a mobile device. For an upcoming study, a mobile application with real-time exergames including markerless motion capture is being developed. However, implementing such technology introduces key challenges, including balancing accuracy and real-time responsiveness, ensuring proper user interaction. Future research should explore optimizing AI models for realtime performance, integrating adaptive gamification, and refining user-centered design principles. By overcoming these challenges, smartphone-based exergames could become powerful tools for engaging users in physical activity and rehabilitation, extending their benefits to a broader audience.

Problem

• Balancing accuracy and real-time responsiveness in smartphone MoCap
• Ensuring proper user interaction in markerless exergames
• Optimizing AI models for real-time performance and accessibility

Innovation

• Markerless MoCap via smartphone cameras
• AI-powered real-time pose estimation
• Mobile exergames for health rehabilitation