Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering

2026-03-14

AI Summary

This work addresses the limitations of existing chain-of-thought (CoT) reasoning in medical visual question answering (VQA), which often adopts unstructured, free-form formats that misalign with clinicians’ systematic diagnostic workflows, thereby compromising both accuracy and interpretability. To bridge this gap, the authors introduce Step-CoT, a large-scale medical reasoning dataset comprising over 10K clinical cases and 70K question-answer pairs, featuring the first structured multi-step reasoning annotations aligned with real-world clinical workflows. They further propose a teacher–student learning framework coupled with a dynamic graph-based focusing mechanism to guide the model along clinically plausible reasoning paths. Experimental results demonstrate that the proposed approach significantly enhances both the reasoning performance and interpretability of medical VQA systems.

Technology Category

• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Reasoning under Uncertainty: Causality

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Chain-of-thought (CoT) reasoning has advanced medical visual question answering (VQA), yet most existing CoT rationales are free-form and fail to capture the structured reasoning process clinicians actually follow. This work asks: Can traceable, multi-step reasoning supervision improve reasoning accuracy and the interpretability of Medical VQA? To this end, we introduce Step-CoT, a large-scale medical reasoning dataset with expert-curated, structured multi-step CoT aligned to clinical diagnostic workflows, implicitly grounding the model's reasoning in radiographic evidence. Step-CoT comprises more than 10K real clinical cases and 70K VQA pairs organized around diagnostic workflows, providing supervised intermediate steps that guide models to follow valid reasoning trajectories. To effectively learn from Step-CoT, we further introduce a teacher-student framework with a dynamic graph-structured focusing mechanism that prioritizes diagnostically informative steps while filtering out less relevant contexts. Our experiments show that using Step-CoT can improve reasoning accuracy and interpretability. Benchmark: github.com/hahaha111111/Step-CoT. Dataset Card: huggingface.co/datasets/fl-15o/Step-CoT

Problem

• Medical Visual Question Answering
• Chain-of-Thought
• Structured Reasoning
• Interpretability
• Clinical Workflow

Innovation

• Stepwise Chain-of-Thought
• Medical Visual Question Answering
• Structured Reasoning
• Teacher-Student Framework
• Graph-Structured Focusing