Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning

2026-04-28

AI Summary

This work addresses the susceptibility of vision-language models (VLMs) to linguistic priors and memory-based prototypes in optical illusion scenarios, which often leads them to overlook genuine visual evidence. To mitigate this, the authors propose a Structured Qualitative Inference (SQI) framework that enhances qualitative understanding and reasoning about visual inputs without fine-tuning the frozen VLM. SQI integrates three synergistic components: axiom-constrained injection to suppress quantitative hallucinations, hierarchical scene disentanglement to isolate background interference, and counterfactual self-verification to overcome confirmation bias. This approach establishes the first purely data-driven qualitative reasoning mechanism, significantly improving the robustness and accuracy of frozen VLMs across diverse optical illusions. The method achieved second place in the DataCV 2026 Challenge and offers high interpretability.

Abstract

While Vision-Language Models (VLMs) have achieved state-of-the-art performance in general visual tasks, their perceptual robustness remains remarkably brittle when confronted with optical illusions. These failures are often attributed to shortcut heuristics, where models prioritize linguistic priors and memorized prototypes over direct visual evidence. In this work, we propose Structured Qualitative Inference (SQI), a training-free, data-centric framework designed to fortify visual grounding in frozen VLMs. SQI addresses perceptual anomalies through three systematic modules: (1) Axiomatic Constraint Injection, which suppresses erroneous metric estimations and quantitative hallucinations; (2) Hierarchical Scene Decomposition, which decouples target visual manifolds from complex background distractors; and (3) Counterfactual Self-Verification, an adversarial reasoning step that mitigates confirmation bias. By orchestrating these qualitative constraints at inference time, SQI effectively aligns high-level linguistic reasoning with low-level visual perception. Our framework was evaluated on the DataCV 2026 Challenge (Task I: Classic Illusion Understanding), where it ranked 2nd place overall. Experimental results demonstrate that SQI not only significantly enhances accuracy across diverse illusion categories but also provides superior diagnostic interpretability without any model fine-tuning. Our success underscores the potential of structured qualitative grounding as a robust paradigm for developing next-generation, illusion-resistant vision-language systems.

Problem

• visual illusions
• Vision-Language Models
• perceptual robustness
• shortcut heuristics
• visual grounding

Innovation

• Structured Qualitative Inference
• Visual Illusions
• Frozen VLMs
• Qualitative Reasoning
• Perceptual Robustness