Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training

2026-04-17

AI Summary

This work addresses the challenges of reward hacking, hallucination, and computational inefficiency in large language models during scientific ideation, which stem from imperfect reward signals. To mitigate these issues, the paper proposes a novel reinforcement learning framework that incorporates the first multi-agent debate mechanism specifically designed for evaluating scientific creativity, delivering strict binary rewards. This approach is combined with an unbiased Group Relative Policy Optimization algorithm for post-training. By decoupling evaluation from implementation, the framework effectively suppresses reward manipulation. Experimental results on the ICLR-320 dataset demonstrate that the proposed method significantly outperforms state-of-the-art baselines across expert-evaluated dimensions of novelty, feasibility, and effectiveness.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) have demonstrated potential in automating scientific ideation, yet current approaches relying on iterative prompting or complex multi-agent architectures often suffer from hallucination or computational inefficiency. A critical bottleneck in applying Reinforcement Learning (RL) to this open-ended domain is reward hacking -- where models exploit imperfect evaluation proxies to maximize scores without producing genuine scientific innovation. To address these limitations, we propose an RL framework explicitly tailored for high-quality scientific idea generation. We propose the first multi-agent reward function designed to serve as a judge, decoupling methodological validation from implementation details while providing strict binary rewards that are robust to reward hacking. To effectively optimize against this sparse signal, we utilize an unbiased variant of Group Relative Policy Optimization to mitigate artificial length bias. We grounded our training in ICLR-320, a curated dataset of problem-solution pairs extracted from ICLR 2024 proceedings. Experiments demonstrate that our framework significantly outperforms state-of-the-art baselines across expert-evaluated metrics of novelty, feasibility, and effectiveness.

Problem

• reward hacking
• scientific ideation
• reinforcement learning
• large language models
• multi-agent reward

Innovation

• multi-agent reward
• reward hacking
• scientific ideation
• reinforcement learning
• Group Relative Policy Optimization