Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

2026-05-12

AI Summary

This work addresses positional bias in multi-image cross-modal retrieval, where the input order inadvertently skews model predictions away from true semantic relevance. The study reveals, for the first time, a significant inconsistency—termed Logit-Attention Divergence—between model output logits and internal attention maps. Leveraging this insight, the authors propose a training-free, inference-time debiasing framework that dynamically corrects predictions at the instance level through attention guidance. This approach transcends conventional logit-only calibration paradigms, achieving effective debiasing with negligible calibration data and zero training overhead. Evaluated on the MS-COCO benchmark, the method substantially enhances model permutation invariance and improves accuracy by over 40% relative to the baseline, establishing a new state-of-the-art performance.

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-Attention Divergence, in which output logits are heavily biased while internal attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40\% compared to baselines. Code is available at https://github.com/brightXian/LAD.

Problem

• position bias
• multi-image retrieval
• multimodal large language models
• logit-attention divergence
• cross-modal retrieval

Innovation

• Logit-Attention Divergence
• position bias
• attention-guided calibration
• training-free debiasing
• multi-image retrieval