CoRe: A Continuously Reward-Finetuned LLM Query Rewriter for Multi-Stage Context-Aware Relevance in Web-Scale Video Search

2026-06-12

AI Summary

This work addresses the challenge in large-scale short-video search where existing large language model–based query rewriting methods struggle to simultaneously achieve low-cost continuous updates and alignment with online ranking objectives. The authors propose a continuously optimized system deployable on a weekly basis, featuring a multiplicative production-aligned reward design, a semi-online hybrid preference optimization mechanism, and an automated reward hijacking detection gate. The approach employs a DPO-style pairwise objective, top-k/bottom-k trajectory sampling, and staged parameter synchronization, integrating rewritten queries as a parallel relevance signal across the entire retrieval-to-reranking pipeline. Two rounds of A/B experiments demonstrate a significant reduction in query alteration rate alongside consistent improvements in core relevance and user engagement metrics, validating the method’s effectiveness and robustness.

Abstract

LLM-based query rewriters in production face a tension: the training reward must reflect how the rewrite is consumed by the production ranker, yet the training procedure must be cheap enough to support continuous redeployment as data drifts. We present CoRe (Context Relevance), such a system, redeployed weekly for over five months in a major short-video search engine. Our reward uses the deployed multimodal relevance model as its source and a multiplicative ratio form mirroring the production fusion algebra, closing the simulation-production gap that offline reward proxies leave open. A semi-online Mixed Preference Optimization loop makes this reward affordable at multi-million-instance weekly scale: a DPO-style pairwise objective restricts the gradient pass to a small top-k/bottom-k subset of sampled trajectories, and a phase structure reduces trainer/inference-server parameter syncs from per-step to per-phase. An automated promotion gate over reward-like and stability metrics detected and recovered from a real reward-hacking incident in production. Rewriter output is consumed as parallel relevance signals at recall, rawrank, and finerank without displacing the original signals, bounding rewriter-failure blast radius. Online A/B from two sequential production launches, first deploying the rewriter at finerank, then extending consumption to recall and rawrank, delivers statistically significant reductions in change-query rate on rewrite-impacted queries, with all headline relevance and engagement metrics moving in the expected direction.

Problem

• query rewriting
• reward alignment
• data drift
• web-scale video search
• production deployment

Innovation

• Continuously Reward-Finetuned LLM
• Context-Aware Query Rewriting
• Mixed Preference Optimization
• Reward-Hacking Mitigation
• Multi-Stage Relevance Integration