VisPlay: Self-Evolving Vision-Language Models from Images

2025-11-19

AI Summary

Existing reinforcement learning approaches for improving vision-language models (VLMs) heavily rely on costly human-annotated labels or task-specific heuristic rewards, limiting scalability and generalization. This paper introduces VisPlay—the first self-evolving VLM training framework leveraging only unlabeled image data. Its core innovation is a role-separation mechanism that decouples a single VLM into interactive “Questioner” and “Reasoner” components, supported by a diversity-difficulty joint reward, image-conditioned question generation, multimodal response modeling, group-wise relative policy optimization (GRPO), silver-answer learning, and self-feedback. Evaluated on Qwen2.5-VL and MiMo-VL, VisPlay achieves consistent improvements across eight mainstream benchmarks, significantly enhancing visual reasoning, compositional generalization, and hallucination mitigation.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reinforcement learning (RL) provides a principled framework for improving Vision-Language Models (VLMs) on complex reasoning tasks. However, existing RL approaches often rely on human-annotated labels or task-specific heuristics to define verifiable rewards, both of which are costly and difficult to scale. We introduce VisPlay, a self-evolving RL framework that enables VLMs to autonomously improve their reasoning abilities using large amounts of unlabeled image data. Starting from a single base VLM, VisPlay assigns the model into two interacting roles: an Image-Conditioned Questioner that formulates challenging yet answerable visual questions, and a Multimodal Reasoner that generates silver responses. These roles are jointly trained with Group Relative Policy Optimization (GRPO), which incorporates diversity and difficulty rewards to balance the complexity of generated questions with the quality of the silver answers. VisPlay scales efficiently across two model families. When trained on Qwen2.5-VL and MiMo-VL, VisPlay achieves consistent improvements in visual reasoning, compositional generalization, and hallucination reduction across eight benchmarks, including MM-Vet and MMMU, demonstrating a scalable path toward self-evolving multimodal intelligence. The project page is available at https://bruno686.github.io/VisPlay/

Problem

• Developing self-evolving vision-language models without human annotations
• Creating autonomous improvement framework for visual reasoning tasks
• Scaling reinforcement learning for multimodal intelligence without costly labels

Innovation

• Self-evolving RL framework for autonomous VLM improvement
• Assigns dual roles for question generation and reasoning
• Uses GRPO with diversity and difficulty rewards