Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search

2026-06-25

AI Summary

This study addresses the challenge in industrial semantic job search where low-bandwidth query interfaces struggle to capture high-dimensional candidate features, resulting in queries with poor generalization and transferability. The authors propose an end-to-end RLAIF framework that leverages classifier-free reinforcement learning algorithms—such as RLOO, REINFORCE++, and GRPO—to generate de-identified job queries preserving general qualifications through AI feedback. A rule-based reward floor mechanism is introduced to suppress reward-hacking behaviors like verbatim copying. Empirical findings reveal that reward shaping exerts a more decisive influence than algorithm choice, with GRPO exhibiting high sensitivity to spurious rewards. Incorporating the reward floor improves cross-model query quality by 0.147 and uncovers a 2.4× overestimation of reward model efficacy during training, underscoring the critical role of reward shaping.

Abstract

Job-search platforms rely on low-bandwidth query interfaces that often fail to capture the high-dimensional complexity of candidate profiles. We present an end-to-end RLAIF (Reinforcement Learning from AI Feedback) framework to generate \emph{portable} job search queries, terms that abstract away seeker-specific identifiers while preserving generalizable qualifications. This task introduces a highly adversarial reward surface where policy optimization frequently exploits flaws in LLM-as-judge rubrics, resulting in degenerate verbatim-copying behaviors. We conducted comprehensive empirical experiments to isolate the impact of optimization mechanics against structured reward engineering. Our results demonstrate that for critic-free optimizers, performance is overwhelmingly dictated by robust reward shaping, rendering the specific choice of algorithm largely immaterial. While critic-free per-rollout baseline methods (RLOO and REINFORCE++) natively resist reward-hacking, the group-relative advantage normalization in GRPO appears uniquely sensitive to spurious reward signals, making it disproportionately susceptible to exploitation. We show that introducing a deterministic, rule-based reward floor to correct for rewards assigned to verbatim copying mitigates this failure mode, resulting in a substantial $+0.147$ quality improvement on a cross-family evaluation judge. Ultimately, we show that the training-time reward model inflates performance gains by $2.4\times$, confirming that the training success is fundamentally dependent on enforcing reward-shaping disciplines rather than selecting alternative optimizers.

Problem

• portable query generation
• reward shaping
• RLAIF
• semantic job search
• reward hacking

Innovation

• RLAIF
• reward shaping
• portable query generation
• reward hacking
• GRPO