SPARK: Synergistic Policy And Reward Co-Evolving Framework

2025-09-26

AI Summary

Existing RLHF relies on costly human preference annotations and suffers from reward-policy misalignment; RLVR, in turn, discards correctness signals inherent in rollouts. This paper proposes SPARK—a computationally efficient framework for co-evolving policy and reward models. SPARK eliminates the need for human annotations or a separate reward model by fine-tuning the language model itself into a generative reward model, thereby establishing a positive feedback loop of joint optimization between policy and reward. Its online reinforcement learning training integrates three complementary objectives: token-level scoring, pairwise comparison, and self-reflective feedback. Comprehensive evaluation across LLMs and LVLMs demonstrates consistent gains: SPARK-VL-7B achieves average improvements of 9.7% on seven reasoning benchmarks, 12.1% on two reward modeling benchmarks, and 1.5% on eight general-purpose benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) increasingly use Reinforcement Learning (RL) for post-pretraining, such as RL with Verifiable Rewards (RLVR) for objective tasks and RL from Human Feedback (RLHF) for subjective tasks. However, RLHF incurs high costs and potential reward-policy mismatch due to reliance on human preferences, while RLVR still wastes supervision by discarding rollouts and correctness signals after each update. To address these challenges, we introduce the Synergistic Policy And Reward Co-Evolving Framework (SPARK), an efficient, on-policy, and stable method that builds on RLVR. Instead of discarding rollouts and correctness data, SPARK recycles this valuable information to simultaneously train the model itself as a generative reward model. This auxiliary training uses a mix of objectives, such as pointwise reward score, pairwise comparison, and evaluation conditioned on further-reflection responses, to teach the model to evaluate and improve its own responses. Our process eliminates the need for a separate reward model and costly human preference data. SPARK creates a positive co-evolving feedback loop: improved reward accuracy yields better policy gradients, which in turn produce higher-quality rollouts that further refine the reward model. Our unified framework supports test-time scaling via self-reflection without external reward models and their associated costs. We show that SPARK achieves significant performance gains on multiple LLM and LVLM models and multiple reasoning, reward models, and general benchmarks. For example, SPARK-VL-7B achieves an average 9.7% gain on 7 reasoning benchmarks, 12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks over the baselines, demonstrating robustness and broad generalization.

Problem

• Addresses RLHF's high costs and reward-policy mismatch issues
• Recycles rollout data to train generative reward models
• Eliminates need for separate reward models and human preferences

Innovation

• Recycles rollouts and correctness data for training
• Trains model as generative reward model using mixed objectives
• Creates co-evolving feedback loop between policy and reward