Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation

2025-06-12

AI Summary

Existing LLM evaluation methods suffer from high API costs, low reliability, inflexible workflows, and inherent biases. To address these limitations, we propose PAJAMA, the first framework to introduce the “Program-as-a-Judge” paradigm: leveraging LLMs to automatically synthesize executable Python judgment programs—replacing opaque, black-box scoring with transparent, interpretable, auditable, and reusable evaluation logic. Key technical contributions include LLM-driven program synthesis, rule distillation, local program execution, and bias calibration. Experiments demonstrate that PAJAMA improves inter-judge consistency by 15.83%, reduces biased responses by 23.7%, outperforms LLM-based judges by 2.19–8.67% on the CHAT-HARD benchmark, and slashes evaluation costs by three orders of magnitude.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA (Program-As-a-Judge for Automated Model Assessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost.

Problem

• High API costs and uncertain reliability in LLM evaluations
• Inflexible pipelines and inherent biases in LLM judging
• Need for interpretable, auditable, and cost-effective evaluation methods

Innovation

• LLMs synthesize executable judging programs
• Local execution reduces costs significantly
• Program-based judges reduce biases effectively