Human-like autonomy emerges from self-play and a pinch of human data

2026-06-11

AI Summary

This work addresses the incompatibility between existing self-play reinforcement learning–derived driving policies and human driving behavior, which hinders effective coordination in real-world traffic despite ensuring safety. The authors propose a method that integrates an extremely small amount of human driving data—only 30 minutes, representing a 2,500-fold reduction compared to typical imitation learning—as a regularization target within a self-play reinforcement learning framework. This approach guides the policy to simultaneously achieve safety and mimic human driving styles, without requiring complex reward engineering or domain randomization. Trained on a single consumer-grade GPU in under 15 hours, the resulting policy demonstrates strong coordination capabilities with unseen human driving trajectories. Code and demonstration videos are publicly released.

Abstract

Self-play reinforcement learning has recently emerged as a way to train driving policies without any human data. It uses cheap, large-scale simulations to substitute expensive, large-scale human driving demonstrations. A key limitation of this approach is that policies trained through pure self-play can learn effective but alien driving conventions incompatible with people. Previous works attempt to mitigate such behavioral misalignments through extensive reward engineering and domain randomization, which are brittle and labor-intensive. Instead of completely discarding human demonstrations, our method treats them as a regularization objective on top of a minimal safe goal-reaching reward. Like the spice in a good stew, we find that a little human data goes a long way: our method uses only 30 minutes of human demonstrations, 2500x fewer than comparable imitation learning approaches. Resulting policies coordinate with held-out human trajectories and complete training in 15 hours on a single consumer-grade GPU. Videos and full source code are available at https://spiced-self-play.com/.

Problem

• self-play
• human-like autonomy
• behavioral alignment
• driving policies
• human demonstrations

Innovation

• self-play reinforcement learning
• human demonstration regularization
• autonomous driving
• behavioral alignment
• sample efficiency