H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

2026-07-01

AI Summary

This work addresses the limitations of existing online policy distillation methods, which rely on static teacher routing and struggle to adapt to the dynamic demands of visual grounding and abstract reasoning during decoding. To overcome this, we propose a confidence-aware heterogeneous multi-teacher online distillation framework that refines teacher arbitration from the sample level to the token level, dynamically integrating the strengths of vision-language and text-only teachers along a shared student trajectory. By combining visual semantic transfer with a confidence-aware mechanism, our approach enables complementary collaboration among heterogeneous teachers during inference. Extensive experiments across eleven mainstream reasoning benchmarks demonstrate significant performance gains over current state-of-the-art methods, confirming the effectiveness and generalization capability of the proposed framework.

Abstract

On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories. However, existing OPD methods for multimodal reasoning usually rely on a static teacher routing, assigning each sample to a single teacher based on modality or task type. This ignores that visual grounding and abstract reasoning may dominate different decoding steps, making a single teacher insufficient for the full trajectory. To this end, H-OPD is proposed as a confidence-aware heterogeneous multi-teacher OPD framework for multimodal reasoning. By verifying the complementarity of heterogeneous teachers in the same reasoning process, H-OPD replaces task or sample level teacher routing with token-level teacher arbitration along the shared student trajectory. H-OPD employs vision-to-language description transfer to enable text-only teachers to access key visual semantics, and uses a confidence-aware arbitration mechanism to dynamically combine vision-language teacher and text-only teachers at each token. Extensive evaluations over 11 widely-used reasoning benchmarks showcase the superior performance of our method.

Problem

• on-policy distillation
• multimodal reasoning
• teacher routing
• heterogeneous teachers
• token-level arbitration

Innovation

• on-policy distillation
• heterogeneous multi-teacher
• token-level arbitration
• confidence-aware fusion
• multimodal reasoning