CoFFT: Chain of Foresight-Focus Thought for Visual Language Models

2025-09-26

AI Summary

Vision-language models (VLMs) suffer from susceptibility to redundant visual information in complex images, leading to task-irrelevant reasoning and hallucination—primarily due to the absence of dynamic, task-guided localization and coordinated attention over salient regions during inference. To address this, we propose CoFFT, a training-free inference enhancement method inspired by human visual cognition. CoFFT establishes a “foresight–focus” chain-of-thought, employing iterative, closed-loop reasoning that integrates multi-step path search, dual-branch visual foresight decoding, and dynamic attention reallocation to precisely calibrate visual focus in a task-driven manner. Evaluated on mainstream VLMs—including Qwen2.5-VL, InternVL-2.5, and LLaVA-Next—CoFFT achieves consistent average improvements of 3.1–5.8% across multiple benchmarks, with negligible computational overhead. The method significantly enhances both robustness and accuracy in complex visual reasoning scenarios.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Despite significant advances in Vision Language Models (VLMs), they remain constrained by the complexity and redundancy of visual input. When images contain large amounts of irrelevant information, VLMs are susceptible to interference, thus generating excessive task-irrelevant reasoning processes or even hallucinations. This limitation stems from their inability to discover and process the required regions during reasoning precisely. To address this limitation, we present the Chain of Foresight-Focus Thought (CoFFT), a novel training-free approach that enhances VLMs' visual reasoning by emulating human visual cognition. Each Foresight-Focus Thought consists of three stages: (1) Diverse Sample Generation: generates diverse reasoning samples to explore potential reasoning paths, where each sample contains several reasoning steps; (2) Dual Foresight Decoding: rigorously evaluates these samples based on both visual focus and reasoning progression, adding the first step of optimal sample to the reasoning process; (3) Visual Focus Adjustment: precisely adjust visual focus toward regions most beneficial for future reasoning, before returning to stage (1) to generate subsequent reasoning samples until reaching the final answer. These stages function iteratively, creating an interdependent cycle where reasoning guides visual focus and visual focus informs subsequent reasoning. Empirical results across multiple benchmarks using Qwen2.5-VL, InternVL-2.5, and Llava-Next demonstrate consistent performance improvements of 3.1-5.8% with controllable increasing computational overhead.

Problem

• VLMs struggle with irrelevant visual information interference
• Models generate excessive task-irrelevant reasoning and hallucinations
• Current VLMs cannot precisely discover required reasoning regions

Innovation

• Training-free approach emulating human visual cognition
• Iterative foresight-focus thought with three stages
• Adjusts visual focus to guide reasoning process