ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval

2026-08-16

AI Summary

This study addresses the limitations of supervised signals in visual document retrieval, specifically their neglect of complex document structures and imprecise evidence representation. We propose ConceptFormer, a framework that learns query-adaptive latent concept representations to bridge local visual evidence and semantic relevance without relying on textual intermediaries or raw annotations. By integrating dynamic token generation guided by strong vision-language models with embedding space optimization, this approach effectively narrows the semantic gap. Experimental results demonstrate that ConceptFormer significantly enhances retrieval performance across multiple benchmarks, achieving average NDCG@10 improvements of 16.7% and 22.1% over state-of-the-art visual and OCR-based text retrieval baselines, respectively. These findings underscore the efficacy of latent concept modeling in advancing visual document retrieval systems beyond conventional supervision paradigms.

Abstract

Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.

Problem

• Visual Document Retrieval
• Query-Document Alignment
• Evidence Representation
• Multimodal RAG

Innovation

• Latent Concept Representation
• Visual Document Retrieval
• Query-Conditioned Tokens
• Adaptive Token Number
• Semantic Alignment