Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration

2025-11-19

AI Summary

Current multimodal reasoning models suffer from static architectures that impede autonomous exploration of diverse reasoning paths and dynamic adaptation to task requirements, leading to fragmented capabilities. To address this, we propose an agent-based multimodal reasoning paradigm. Our method integrates autonomous agent architecture, dynamic capability scheduling, tool-augmented visual exploration, programmatic image manipulation, and intrinsic visual imagination. We systematically define six core reasoning capabilities and introduce Octopus-Bench—a comprehensive, multi-dimensional evaluation benchmark—to rigorously assess multimodal reasoning. Furthermore, we design a state-aware capability coordination mechanism to enable adaptive, synergistic execution across reasoning modalities. On Octopus-Bench, our approach achieves state-of-the-art performance across most tasks, demonstrating that coordinated capability integration significantly enhances reasoning flexibility, task adaptability, and robustness in real-world scenarios.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Existing multimodal reasoning models and frameworks suffer from fundamental architectural limitations: most lack the human-like ability to autonomously explore diverse reasoning pathways-whether in direct inference, tool-driven visual exploration, programmatic visual manipulation, or intrinsic visual imagination. Consequently, they struggle to adapt to dynamically changing capability requirements in real-world tasks. Meanwhile, humans exhibit a complementary set of thinking abilities when addressing such tasks, whereas existing methods typically cover only a subset of these dimensions. Inspired by this, we propose Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration, a new paradigm for multimodal agentic reasoning. We define six core capabilities essential for multimodal reasoning and organize a comprehensive evaluation benchmark, Octopus-Bench, accordingly. Octopus is capable of autonomously exploring during reasoning and dynamically selecting the most appropriate capability based on the current state. Experimental results show that Octopus achieves the best performance on the vast majority of tasks in Octopus-Bench, highlighting the crucial role of capability coordination in agentic multimodal reasoning.

Problem

• Existing models lack autonomous exploration of reasoning pathways
• Current frameworks struggle with dynamic capability adaptation
• Human-like multimodal reasoning requires coordinated six-capability orchestration

Innovation

• Autonomously explores diverse reasoning pathways dynamically
• Orchestrates six core capabilities for multimodal reasoning
• Dynamically selects most appropriate capability based on state