A Champion-Level Vision-Based Reinforcement Learning Agent for Competitive Racing in Gran Turismo 7

2025-04-12

IEEE Robotics and Automation Letters

AI Summary

This study addresses the challenge of achieving high-performance autonomous racing using only onboard camera and sensor data—without external positioning systems—under realistic conditions. Method: We propose the first end-to-end vision-driven reinforcement learning racing agent, trained in the high-fidelity Gran Turismo 7 simulator. Our approach features an asymmetric Actor-Critic architecture: the Actor operates solely on local sensory inputs, while the Critic leverages global information during training to improve policy quality. The agent integrates an RNN-based policy network, a lightweight visual encoder, a self-supervised trajectory prediction module, and an asynchronous distributed PPO framework. Results: The agent consistently outperforms all built-in GT7 AI opponents and achieves performance comparable to professional human racers. Its core contribution is the empirical validation of high-fidelity autonomous racing feasibility using purely onboard perception, establishing a transferable technical paradigm for real-world track-based autonomous driving.

Technology Category

• Computer Vision: Vision for Robotics & Autonomous Driving
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

Deep reinforcement learning has achieved superhuman racing performance in high-fidelity simulators like Gran Turismo 7 (GT7). It typically utilizes global features that require instrumentation external to a car, such as precise localization of agents and opponents, limiting real-world applicability. To address this limitation, we introduce a vision-based autonomous racing agent that relies solely on ego-centric camera views and onboard sensor data, eliminating the need for precise localization during inference. This agent employs an asymmetric actor-critic framework: the actor uses a recurrent neural network with the sensor data local to the car to retain track layouts and opponent positions, while the critic accesses the global features during training. Evaluated in GT7, our agent consistently outperforms GT7's built-drivers. To our knowledge, this work presents the first vision-based autonomous racing agent to demonstrate champion-level performance in competitive racing scenarios.

Problem

• Develops vision-based agent for autonomous racing without external instrumentation
• Uses ego-centric camera views and onboard sensors for real-world applicability
• Achieves champion-level performance in competitive racing scenarios

Innovation

• Vision-based agent using ego-centric camera views
• Asymmetric actor-critic framework with recurrent network
• Champion-level performance without precise localization