Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

2026-06-22

AI Summary

Traditional trajectory imitation is confined to replicating specific solution steps and struggles to transfer general reasoning capabilities. This work proposes Strategy-Guided Policy Optimization (SGPO), which elevates instance-level imitation to strategy-level distillation by extracting structured strategy descriptions from a strong teacher model. SGPO generates paired trajectories—with and without strategy guidance—and integrates a selective forward KL objective, proximal constraint optimization, and an adaptive instance weighting mechanism to enable efficient and stable knowledge transfer. Evaluated on four mathematical reasoning benchmarks, SGPO substantially outperforms standard supervised fine-tuning and online policy reinforcement learning baselines, achieving an average improvement of 2.2 points on Qwen2.5-7B-Instruct, thereby demonstrating its effectiveness and scalability.

Abstract

Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason. This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation. SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model's behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model's own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability.

Problem

• reasoning distillation
• trajectory imitation
• generalization
• strategy learning
• language models

Innovation

• Strategy Distillation
• Forward-KL Objective
• Adaptive Weighting
• Policy Optimization
• Reasoning Transfer