Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuning

2026-04-17

AI Summary

This work addresses the vulnerability of multimodal large language models to endogenous reasoning drift during reinforcement fine-tuning, which induces unpredictable distributional shifts in autoregressive generation and compromises reasoning consistency and decision reliability. The study formalizes this issue as multimodal concept drift and introduces CPO++, a novel framework that integrates counterfactual reasoning with domain knowledge to enable autonomous optimization. By decoupling spurious correlations through controlled perception–reasoning perturbations and preference optimization, CPO++ achieves end-to-end adaptation in non-stationary environments. Empirical results demonstrate that the proposed method substantially enhances reasoning coherence, decision accuracy, and robustness against interference in high-dynamic, safety-critical scenarios such as medical diagnosis and autonomous driving, while also exhibiting exceptional zero-shot cross-domain generalization performance.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement Fine-Tuning (RFT) has established itself as a critical paradigm for the alignment of Multi-modal Large Language Models (MLLMs) with complex human values and domain-specific requirements. Nevertheless, current research primarily focuses on mitigating exogenous distribution shifts arising from data-centric factors, the non-stationarity inherent in the endogenous reasoning remains largely unexplored. In this work, a critical vulnerability is revealed within MLLMs: they are highly susceptible to endogenous reasoning drift, across both thinking and perception perspectives. It manifests as unpredictable distribution changes that emerge spontaneously during the autoregressive generation process, independent of external environmental perturbations. To adapt it, we first theoretically define endogenous reasoning drift within the RFT of MLLMs as the multi-modal concept drift. In this context, this paper proposes Counterfactual Preference Optimization ++ (CPO++), a comprehensive and autonomous framework adapted to the multi-modal concept drift. It integrates counterfactual reasoning with domain knowledge to execute controlled perturbations across thinking and perception, employing preference optimization to disentangle spurious correlations. Extensive empirical evaluations across two highly dynamic and safety-critical domains: medical diagnosis and autonomous driving. They demonstrate that the proposed framework achieves superior performance in reasoning coherence, decision-making precision, and inherent robustness against extreme interference. The methodology also exhibits exceptional zero-shot cross-domain generalization, providing a principled foundation for reliable multi-modal reasoning in safety-critical applications.

Problem

• endogenous reasoning drift
• multi-modal large language models
• non-stationary tuning
• concept drift
• reinforcement fine-tuning

Innovation

• endogenous reasoning drift
• multi-modal concept drift
• Counterfactual Preference Optimization++
• reinforcement fine-tuning
• zero-shot cross-domain generalization