From Gameplay to Policy: Towards Scalable Robot Data Collection via Gamified Robot-Free Interaction

2026-09-16

AI Summary

为解决机器人操作数据收集成本高、多样性不足的问题，提出了一种基于VR游戏的交互平台Project Kitchen，并通过Game2Policy方法将游戏中的操作经验转换为机器人策略，提高了学习效率和成功率。

Abstract

Learning generalizable robot manipulation policies requires large-scale and diverse interaction data, yet collecting real-world demonstrations remains costly and difficult to scale. Existing approaches to data collection are either dependent on specific robot hardware that limits crowdsourcing and transferability, or suffer from incomplete annotation and limited behavioral diversity. Inspired by how games sustain long-term human engagement, we explore an alternative paradigm that turns data collection into an engaging gameplay experience and transfers the resulting human manipulation experience to real robots. We present Project Kitchen, a VR-based gamified egocentric data collection platform that elicits diverse, goal-directed manipulation while remaining independent of specific robot embodiments and hardware, making it applicable to broader and potentially large-scale deployment. To bridge the game-to-real gap, we further introduce Game2Policy, which extracts embodiment-invariant affordance cues, including contact points and sub-goal states, from gameplay trajectories. An affordance model is pre-trained on game-collected data and then jointly fine-tuned with downstream policies using only a handful of real-robot demonstrations. Experiments show that Game2Policy improves average success rates by 10.0 points in simulation and 18.3 points on real robots in the few-shot setting. User studies and quantitative analyses further show that Project Kitchen promotes diverse manipulation behaviors and provides an engaging data collection experience. These results demonstrate the potential of gamified virtual environments as a scalable source of manipulation knowledge. The platform and code will be released upon acceptance.

Problem

• robot manipulation
• data collection
• scalability
• behavioral diversity
• embodiment-invariant

Innovation

• gamified data collection
• embodiment-invariant affordance cues
• few-shot learning
• virtual reality (VR)