Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation

2026-04-29

AI Summary

This work identifies an architectural flaw in hybrid reasoning language models, where shared feedforward network parameters between reasoning and non-reasoning modes lead to redundant, self-referential outputs in the latter. To address this, the authors propose a path-locking mechanism that introduces two semantically locked expert MLPs in each Transformer decoder layer, with a control token deterministically routing computation through a single path, thereby fully decoupling the feedforward pathways. The model retains shared attention and embedding layers to preserve dense computation and undergoes supervised fine-tuning for mode purification. Evaluated on Qwen3-4B, this approach reduces self-referential tokens in non-reasoning mode from 2.54 to 0.39 and improves accuracy from 20.67% to 40.00%, while maintaining reasoning-mode performance.

Abstract

Hybrid-thinking language models expose explicit think and no-think modes, but current designs do not separate them cleanly. Even in no-think mode, models often emit long and self-reflective responses, causing reasoning leakage. Existing work reduces this issue through better data curation and multi-stage training, yet leakage remains because both modes are still encoded in the same feed-forward parameters. We propose Path-Lock Expert (PLE), an architecture-level solution that replaces the single MLP in each decoder layer with two semantically locked experts, one for think and one for no-think, while keeping attention, embeddings, normalization, and the language-model head shared. A deterministic control-token router selects exactly one expert path for the entire sequence, so inference preserves the dense model's per-token computation pattern and each expert receives mode-pure updates during supervised fine-tuning. Across math and science reasoning benchmarks, PLE maintains strong think performance while producing a substantially stronger no-think mode that is more accurate, more concise, and far less prone to reasoning leakage. On Qwen3-4B, for example, PLE reduces no-think reflective tokens on AIME24 from 2.54 to 0.39 and improves no-think accuracy from 20.67% to 40.00%, all while preserving think-mode performance. These results suggest that controllable hybrid thinking is fundamentally an architectural problem, and separating mode-specific feed-forward pathways is a simple and effective solution.

Problem

• reasoning leakage
• hybrid thinking
• think/no-think modes
• architecture-level separation
• language models

Innovation

• Path-Lock Expert
• hybrid thinking
• reasoning leakage
• architecture-level separation
• mode-specific experts