TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding

2026-09-07

AI Summary

为解决现有视觉语言模型在细粒度视觉感知上的不足，提出TDDN网络，通过融合DINOv3和CleanDIFT表征并结合文本对齐方法，提高图像理解能力。

Abstract

Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only $\sim$590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 $\to$ 18.11 mIoU, COCO-Stuff 7.35 $\to$ 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP$\,$2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 $\to$ 22.51 mIoU).

Problem

• Structured visual reasoning
• fine-grained visual perception
• Vision Language Models (VLMs)
• CLIP-based ViT backbones

Innovation

• TDDN
• fine-grained visual perception
• DINOv3 and CleanDIFT fusion
• text-alignment
• Puzzle Perception