From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasoners

2026-09-11

AI Summary

研究通过协作内化方法，使用Rivet系统整合专家决策和控制器决策来提高紧凑推理器在移除专家后的生成能力。

Abstract

A compact controller can coordinate stronger experts by selecting whom to consult, formulating requests, and integrating their responses. We study whether learning from both the controller's decisions and the experts' reasoning and code improves its generation after expert removal. We introduce \textsc{Rivet} for \emph{collaboration internalization}: expert-augmented reinforcement learning applies a shared outcome signal to controller decisions and returned expert spans, and verified trajectory internalization consolidates complete successful interactions through format-aware supervised training. The deployed controller generates reasoning, code, and interaction structure with local Python execution and no external LLM. Across seven competition-mathematics benchmarks, RIVET-1.7B and RIVET-4B achieve average accuracies of $28.25\%$ and $44.16\%$; Stage~II improves RIVET-4B's accuracy after expert removal by $6.49$ points, and GPQA-Diamond results provide evidence of generalization to scientific reasoning. Ablations show gains from ordinary trajectory supervision and additional format weighting, supporting the effectiveness of training on the content and structure of verified collaborations.

Problem

• compact controller
• expert reasoning
• generation improvement
• expert removal
• reinforcement learning

Innovation

• expert-augmented reinforcement learning
• verified trajectory internalization
• compact controller
• local Python execution
• collaboration internalization