PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data

2026-07-06

AI Summary

This work addresses a key limitation of current large language models (LLMs) as AI evaluators—their reliance on aggregate consensus while neglecting individual judgment variability. The study presents the first systematic exploration of simulating personalized human preferences by integrating evaluator-specific auxiliary data, such as chain-of-thought reasoning traces and interface telemetry, with in-context learning. Findings reveal that a neutral usage tendency emerges as a stable and predictable cross-task indicator of individual preference. The proposed approach achieves up to a 9.9-percentage-point improvement over baseline methods, with reasoning traces contributing the largest performance gain, whereas interface telemetry often degrades accuracy. These results highlight both the promise and inherent constraints of personalizing LLM-based evaluation.

Abstract

Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a novel simulation approach that combines categorical judgments with evaluator-specific auxiliary data--retrospective reasoning traces and interface telemetry--to enable LLM-based simulation of individual evaluators via in-context learning. We conduct a systematic empirical study of this approach using multi-facet data from 32 trained annotators across 4,200 preference judgments in a 4 x 4 x 4 factorial design. Our key findings: (1) The simulation approach achieves up to 9.9 percentage point improvements over the Base Judge; (2) Reasoning traces provide the largest gains with higher collection efforts, while interface telemetry often hurts rather than helps performance despite being cheaper to collect. (3) Simulation difficulty is systematic, predicted by an evaluator's neutral usage (most clearly on Helpfulness) and divergence from consensus; the neutral-usage tendency--rather than simulatability itself--is the cross-task-stable property (r = 0.728). These results establish both the potential and limits of evaluator-specific auxiliary data for personalized evaluation, offering methodological insights for scaling individual aware AI assessment.

Problem

• individual preference
• LLM-based evaluation
• evaluator variation
• personalized judgment
• preference simulation

Innovation

• individual preference simulation
• evaluator-specific data
• reasoning traces
• in-context learning
• personalized AI evaluation