Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization

2026-04-22

AI Summary

This study investigates how automated prompt optimization can enhance the effectiveness of large language models (LLMs) as evaluators in free-text legal question answering and examines the impact of evaluator bias on prompt generalization. Leveraging the LEXam benchmark, the authors employ the ProTeGi method with either lenient or strict LLM evaluators (Qwen3-32B and DeepSeek-V3) to optimize prompts and assess cross-evaluator transfer performance across four task models. Results demonstrate that automatically optimized prompts significantly outperform handcrafted ones. Moreover, prompts generated by lenient evaluators exhibit superior generalization and stability, and transfer more effectively to strict evaluators than vice versa, highlighting the critical role of evaluator bias in prompt optimization and generalization.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Evaluation and Analysis
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This work explores the role of prompt design and judge selection in LLM-as-a-Judge evaluations of free text legal question answering. We examine whether automatic task prompt optimization improves over human-centered design, whether optimization effectiveness varies by judge feedback style, and whether optimized prompts transfer across judges. We systematically address these questions on the LEXam benchmark by optimizing task prompts using the ProTeGi method with feedback from two judges (Qwen3-32B, DeepSeek-V3) across four task models, and then testing cross-judge transfer. Automatic optimization consistently outperforms the baseline, with lenient judge feedback yielding higher and more consistent gains than strict judge feedback. Prompts optimized with lenient feedback transfer better to strict judges than the reverse direction. Analysis reveals that lenient judges provide permissive feedback, yielding prompts with broader applicability, whereas strict judges produce restrictive feedback, leading to judge-specific overfitting. Our findings demonstrate algorithmically optimizing prompts on training data can outperform human-centered prompt design and that judges' dispositions during optimization shape prompt generalizability. Code and optimized prompts are available at https://github.com/TUMLegalTech/icail2026-llm-judge-gaming.

Problem

• LLM-as-a-Judge
• prompt optimization
• legal question answering
• judge feedback style
• prompt transfer

Innovation

• LLM-as-a-Judge
• prompt optimization
• legal QA
• judge disposition
• cross-judge transfer