SAGE: Scalable AI Governance&Evaluation

2026-02-08

AI Summary

This work addresses the challenge of detecting high-impact relevance failures in large-scale search systems, where limited human annotation resources render conventional user-behavior-based or sparse-review approaches insufficient. To this end, we propose SAGE, a novel framework that introduces a co-evolutionary calibration mechanism among policies, precedents, and large language model (LLM)-based proxy judges. This mechanism transforms subjective relevance judgments into actionable, multi-dimensional scoring criteria and enables cost-effective, scalable deployment through teacher–student distillation. By systematically resolving semantic ambiguities, SAGE generates high-quality, scalable evaluation signals. Deployment in LinkedIn’s search system demonstrates that SAGE effectively identifies model regressions undetectable by traditional metrics, resulting in a 0.25% increase in daily active users.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Evaluating relevance in large-scale search systems is fundamentally constrained by the governance gap between nuanced, resource-constrained human oversight and the high-throughput requirements of production systems. While traditional approaches rely on engagement proxies or sparse manual review, these methods often fail to capture the full scope of high-impact relevance failures. We present \textbf{SAGE} (Scalable AI Governance \&Evaluation), a framework that operationalizes high-quality human product judgment as a scalable evaluation signal. At the core of SAGE is a bidirectional calibration loop where natural-language \emph{Policy}, curated \emph{Precedent}, and an \emph{LLM Surrogate Judge} co-evolve. SAGE systematically resolves semantic ambiguities and misalignments, transforming subjective relevance judgment into an executable, multi-dimensional rubric with near human-level agreement. To bridge the gap between frontier model reasoning and industrial-scale inference, we apply teacher-student distillation to transfer high-fidelity judgments into compact student surrogates at \textbf{92$\times$} lower cost. Deployed within LinkedIn Search ecosystems, SAGE guided model iteration through simulation-driven development, distilling policy-aligned models for online serving and enabling rapid offline evaluation. In production, it powered policy oversight that measured ramped model variants and detected regressions invisible to engagement metrics. Collectively, these drove a \textbf{0.25\%} lift in LinkedIn daily active users.

Problem

• relevance evaluation
• AI governance
• human oversight
• large-scale search systems
• evaluation gap

Innovation

• Scalable AI Governance
• LLM Surrogate Judge
• Policy-Precedent Calibration
• Teacher-Student Distillation
• Relevance Evaluation