Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking

2025-04-04

AI Summary

Small-scale language models (e.g., 3B-parameter) exhibit weak reasoning capabilities in document re-ranking and heavily rely on manual annotations or large teacher models. Method: This paper proposes a lightweight training paradigm integrating knowledge distillation with Proximal Policy Optimization (PPO)-based reinforcement learning. It innovatively formulates re-ranking as a sequential decision-making problem, guiding the student model to explicitly generate relevance explanations—rather than scalar scores—enabling chain-of-reasoning-driven ranking. Explanatory web data are auto-generated by large models, and generalization is enhanced via a teacher–student architecture with self-supervised explanation distillation. Contribution/Results: The approach achieves state-of-the-art performance on the BRIGHT benchmark, ranking third on the leaderboard—outperforming teacher models with over 20× more parameters—while delivering high accuracy, strong interpretability, and efficient deployment.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present a novel approach for training small language models for reasoning-intensive document ranking that combines knowledge distillation with reinforcement learning optimization. While existing methods often rely on expensive human annotations or large black-box language models, our methodology leverages web data and a teacher LLM to automatically generate high-quality training examples with relevance explanations. By framing document ranking as a reinforcement learning problem and incentivizing explicit reasoning capabilities, we train a compact 3B parameter language model that achieves state-of-the-art performance on the BRIGHT benchmark. Our model ranks third on the leaderboard while using substantially fewer parameters than other approaches, outperforming models that are over 20 times larger. Through extensive experiments, we demonstrate that generating explanations during inference, rather than directly predicting relevance scores, enables more effective reasoning with smaller language models. The self-supervised nature of our method offers a scalable and interpretable solution for modern information retrieval systems.

Problem

• Improving document re-ranking with small language models
• Reducing reliance on human annotations and large models
• Enhancing reasoning capabilities for better retrieval performance

Innovation

• Knowledge distillation with reinforcement learning optimization
• Automatically generating training examples using web data and teacher LLM
• Training compact 3B parameter model with explicit reasoning capabilities