J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

2025-05-15

AI Summary

To address the limitations of large language models (LLMs) as evaluators—namely, weak reasoning capabilities and susceptibility to bias—this paper proposes J1, a reinforcement learning training framework. J1 formalizes multimodal prompts as verifiable, reward-driven structured judgment tasks. It introduces the first verifiable reward mechanism specifically designed for evaluation tasks, enabling reasoning incentives even for non-verifiable prompts. Crucially, it provides the first empirical validation that “self-referential contrast” and “explicit modeling of evaluation criteria” significantly enhance judgment quality. Methodologically, J1 integrates PPO optimization, chain-of-thought distillation, pairwise/pointwise architectures, and online-offline hybrid training. Comprehensive evaluations across 8B and 70B model scales demonstrate consistent superiority over state-of-the-art evaluator models, including distilled variants of DeepSeek-R1. Notably, sub-8B variants surpass o1-mini and certain R1 variants, achieving SOTA performance on multiple authoritative evaluation benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The progress of AI is bottlenecked by the quality of evaluation, and powerful LLM-as-a-Judge models have proved to be a core solution. Improved judgment ability is enabled by stronger chain-of-thought reasoning, motivating the need to find the best recipes for training such models to think. In this work we introduce J1, a reinforcement learning approach to training such models. Our method converts both verifiable and non-verifiable prompts to judgment tasks with verifiable rewards that incentivize thinking and mitigate judgment bias. In particular, our approach outperforms all other existing 8B or 70B models when trained at those sizes, including models distilled from DeepSeek-R1. J1 also outperforms o1-mini, and even R1 on some benchmarks, despite training a smaller model. We provide analysis and ablations comparing Pairwise-J1 vs Pointwise-J1 models, offline vs online training recipes, reward strategies, seed prompts, and variations in thought length and content. We find that our models make better judgments by learning to outline evaluation criteria, comparing against self-generated reference answers, and re-evaluating the correctness of model responses.

Problem

• Improving LLM-as-a-Judge evaluation quality via reinforcement learning
• Training models to enhance chain-of-thought reasoning for better judgments
• Mitigating judgment bias by incentivizing thinking with verifiable rewards

Innovation

• Reinforcement learning for LLM-as-a-Judge training
• Converts prompts to verifiable judgment tasks
• Outlines criteria and compares self-generated answers