ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representations

2025-05-20

AI Summary

Existing VI-CoT evaluation benchmarks rely on fixed intermediate visual states (IVS), distorting authentic reasoning trajectories and lacking systematic analysis of IVS influence mechanisms. To address this, we propose ViC-Bench—the first multi-task benchmark enabling free-form IVS generation and evaluation across maze navigation, jigsaw solving, embodied long-horizon planning, and complex counting. Our method introduces a free-form IVS generation pipeline, function-call-driven IVS modeling, a three-stage progressive evaluation framework, and an incremental prompt information injection (IPII) strategy to decouple the impact of individual prompt components on VI-CoT performance. We conduct large-scale evaluation across 18 state-of-the-art multimodal large language models (MLLMs), uncovering critical bottlenecks in their VI-CoT capabilities. ViC-Bench is publicly released on Hugging Face, establishing a new paradigm for interpretable and evolvable visual reasoning assessment.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Visual-Interleaved Chain-of-Thought (VI-CoT) enables MLLMs to continually update their understanding and decisions based on step-wise intermediate visual states (IVS), much like a human would, which demonstrates impressive success in various tasks, thereby leading to emerged advancements in related benchmarks. Despite promising progress, current benchmarks provide models with relatively fixed IVS, rather than free-style IVS, whch might forcibly distort the original thinking trajectories, failing to evaluate their intrinsic reasoning capabilities. More importantly, existing benchmarks neglect to systematically explore the impact factors that IVS would impart to untamed reasoning performance. To tackle above gaps, we introduce a specialized benchmark termed ViC-Bench, consisting of four representive tasks: maze navigation, jigsaw puzzle, embodied long-horizon planning, and complex counting, where each task has dedicated free-style IVS generation pipeline supporting function calls. To systematically examine VI-CoT capability, we propose a thorough evaluation suite incorporating a progressive three-stage strategy with targeted new metrics. Besides, we establish Incremental Prompting Information Injection (IPII) strategy to ablatively explore the prompting factors for VI-CoT. We extensively conduct evaluations for 18 advanced MLLMs, revealing key insights into their VI-CoT capability. Our proposed benchmark is publicly open at Huggingface.

Problem

• Evaluating MLLMs' reasoning with free-style visual states
• Assessing impact of intermediate states on reasoning performance
• Benchmarking visual-interleaved chain-of-thought in diverse tasks

Innovation

• Free-style IVS generation pipeline for diverse tasks
• Progressive three-stage evaluation suite with new metrics
• Incremental Prompting Information Injection (IPII) strategy