Problem
Research questions and friction points this paper is trying to address.
LLM-as-a-Judge
lifecycle
large-scale recommendation explanations
production system
continuous maintenance
The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
🤖 AI Summary
本文探讨了LLM作为评判者在大规模推荐解释中的生命周期问题，通过构建、训练、部署和持续维护四个阶段解决其在生产系统中面临的挑战。
📝 Abstract
LLM-as-a-Judge, which leverages a large language model to evaluate natural language generated by another AI application or model, has become a standard, scalable approach for accelerating and extending costly human evaluation. However, most work treats a judge as a static artifact, evaluating it once at construction or against a fixed benchmark. In contrast, we argue that an LLM judge running in a production system is better understood as having a lifecycle: it must be built, trained, deployed, and continuously maintained as the surrounding data evolves, and each phase poses distinct technical and operational challenges.
We present such a lifecycle for the LLM judges that evaluate user-facing recommendation explanations at Netflix, where our pipeline generates and the judges assess hundreds of thousands of distinct show-level explanations per week, served across the mobile experience to millions of members. Our framework has four phases: (I) Birth, defining multiple evaluation criteria and building curated benchmark datasets with human labels and rationales; (II) Training, refining the judges' rubrics via Reasoning-Aligned Rubric Tuning (RART), a rubric-tuning procedure that uses a meta-judge over reasoning output as the learning signal; (III) Deployment, in which one judge serves two production roles: quality gating and reflective generation; and (IV) Monitoring, a continuous Human-in-the-Loop alignment process that detects drift and triggers re-tuning behind a human review gate. We report post-launch results from a five-week A/B test over tens of millions of members, in which the judge-aligned explanations shifted member viewing toward novel content (previously unwatched) and increased successful browse-to-play sessions relative to a no-explanation control, with no quality-related takedowns.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Lifecycle of LLM-as-a-Judge
Reasoning-Aligned Rubric Tuning (RART)
Continuous Human-in-the-Loop Alignment
🔎 Similar Papers
No similar papers found.
