Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

2026-07-06

AI Summary

This work addresses the challenge that multimodal large language models often struggle to accurately localize targets in visually dense tasks due to their neglect of structured relationships among objects. To overcome this limitation, the authors propose the “Scene Graph Reasoning” (SaGe) paradigm, which explicitly integrates scene graphs into multimodal reasoning for the first time. An automated data engine constructs hierarchical scene graphs with relational edges from image–text corpora and samples 120,000 structured reasoning trajectories. The model is then trained via a two-stage graph alignment strategy: first, supervised fine-tuning internalizes structured reasoning capabilities, followed by reinforcement fine-tuning that introduces a node-agent reward mechanism to optimize graph exploration efficiency. The approach achieves significant performance gains across eight mainstream benchmarks, particularly excelling in fine-grained visual perception and reasoning tasks.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.

Problem

• Multimodal Large Language Models
• structured visual reasoning
• scene graph
• visual relationships
• fine-grained perception

Innovation

• Scene Graph
• Multimodal Large Language Models
• Structured Visual Reasoning
• Graph-Aligned Training
• Reinforcement Fine-Tuning