LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs

2025-01-10

AI Summary

Current large language models lack systematic evaluation protocols and stepwise reasoning capabilities for multi-step visual reasoning tasks. To address this, we propose the first comprehensive framework specifically designed for multi-step visual reasoning. Our contributions include: (1) a new benchmark comprising over 4,000 human-annotated reasoning steps across eight complex task categories; (2) the first fine-grained, step-level evaluation metric that jointly assesses step correctness and logical coherence; and (3) LlamaV-o1, a novel multimodal model integrating multi-step curriculum learning, vision-language joint representation learning, and an interpretable hierarchical reasoning mechanism. Experimental results demonstrate that LlamaV-o1 achieves a mean accuracy of 67.3% across six benchmarks—outperforming LLaVA-CoT by 3.8 percentage points—and attains a 5× speedup in inference latency, consistently surpassing leading open-source models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reasoning is a fundamental capability for solving complex multi-step problems, particularly in visual contexts where sequential step-wise understanding is essential. Existing approaches lack a comprehensive framework for evaluating visual reasoning and do not emphasize step-wise problem-solving. To this end, we propose a comprehensive framework for advancing step-by-step visual reasoning in large language models (LMMs) through three key contributions. First, we introduce a visual reasoning benchmark specifically designed to evaluate multi-step reasoning tasks. The benchmark presents a diverse set of challenges with eight different categories ranging from complex visual perception to scientific reasoning with over 4k reasoning steps in total, enabling robust evaluation of LLMs' abilities to perform accurate and interpretable visual reasoning across multiple steps. Second, we propose a novel metric that assesses visual reasoning quality at the granularity of individual steps, emphasizing both correctness and logical coherence. The proposed metric offers deeper insights into reasoning performance compared to traditional end-task accuracy metrics. Third, we present a new multimodal visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum learning approach, where tasks are progressively organized to facilitate incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is designed for multi-step reasoning and learns step-by-step through a structured training paradigm. Extensive experiments show that our LlamaV-o1 outperforms existing open-source models and performs favorably against close-source proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an average score of 67.3 with an absolute gain of 3.8% across six benchmarks while being 5 times faster during inference scaling. Our benchmark, model, and code are publicly available.

Problem

• Multistep Visual Reasoning
• Large Language Models
• Testing Framework

Innovation

• Multi-step Problem Solving
• Visual Reasoning Assessment
• LlamaV-o1 Model