MMDynOpt-Agent: Dynamic Optimization for Multimodal Large Language Model Reasoning via Reinforcement Learning

2026-08-14

AI Summary

This study addresses the inefficient alignment between visual cues and question semantics in multimodal large language models by formulating multimodal reasoning as a Markov Decision Process. We propose a reinforcement learning-based dynamic prompt optimization framework that employs a lightweight agent for end-to-end training. A budget-aware composite reward mechanism is designed to balance accuracy and efficiency, adaptively guiding the target model through multi-turn reasoning. Experimental results across 15 datasets demonstrate that our method significantly outperforms existing baselines, exhibiting strong generalization and cross-model transferability. Ultimately, this approach effectively achieves both efficient and precise multimodal reasoning, overcoming critical limitations in current semantic translation processes within multimodal systems.

Abstract

Recently, multimodal large language models (MLLMs) have demonstrated strong potential in visual understanding and complex reasoning tasks. However, existing methods often struggle to efficiently transform visual cues from multimodal inputs and the semantics of the question into effective reasoning conditions, thereby limiting the reasoning performance of multimodal large language models. To address this challenge, we propose MMDynOpt-Agent, which models the dynamic optimization of multimodal reasoning as a Markov decision process via end-to-end reinforcement learning. Specifically, a lightweight multimodal agent serves as the decision policy and interacts with the target MLLM as the environment, adaptively steering its reasoning through multi-turn dynamic optimization prompts. Furthermore, to reduce the cost of multimodal reasoning, a reward mechanism that combines format compliance, answer correctness, and budget awareness is designed to jointly ensure reasoning accuracy and efficiency. MMDynOpt-Agent is transferable and generalizable, enabling training with one target MLLM and inference-time transfer to others. Experimental results on fifteen public datasets show MMDynOpt-Agent achieves strong performance and outperforms baselines. Our project is available at https://github.com/QwenQKing/MMDynOpt-Agent.

Problem

• Multimodal Large Language Models
• Reasoning Performance
• Visual Cues Transformation
• Reasoning Efficiency

Innovation

• Reinforcement Learning
• Dynamic Optimization
• Multimodal Reasoning
• Budget-aware Reward
• Transferable Agent