Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation

2026-06-18

AI Summary

This work addresses the prevalent issue of object hallucination in vision-language models (VLMs), where models generate descriptions of objects absent from input images. The authors propose QK Product Steering, a novel method that decomposes the query-key product in attention mechanisms into symmetric and antisymmetric components. They discover that hallucination signals predominantly reside in the symmetric cross-attention channels and leverage this insight to design a closed-form weight-editing strategy that requires no additional data, training, or inference overhead. By updating only the query projection weights to suppress dominant singular modes, the approach effectively mitigates object hallucination. Compatible with grouped-query attention (GQA), the method reduces the CHAIR$_s$ metric by 4.0% on average across three GQA-based VLMs while preserving general multimodal capabilities.

Abstract

Vision-language models (VLMs) often generate fluent but visually unsupported descriptions, especially by mentioning objects absent from the image. We propose QK Product Steering, a data-free, training-free, and zero-inference-cost weight edit for reducing object hallucination. The method directly edits the per-head query-key product, the operator that produces pre-softmax attention logits, by suppressing a small number of dominant singular modes in selected middle layers. The edited product is then mapped back to the query weights through a closed-form query-only update while keeping shared key weights fixed, making the edit compatible with grouped-query attention. We further decompose the QK product into symmetric and antisymmetric components to distinguish mutual content-similarity patterns from directional attention patterns. Across three GQA-based VLMs, QK Product Steering achieves an average relative CHAIR$_s$ reduction of $4.0\%$, while matched random-mode controls show negligible change. Interpretability ablations show that the hallucination signal is specific to dominant QK modes and is primarily localized to the symmetric mutual-attention channel. Overall, QK Product Steering offers a simple alternative to decoding-time mitigation, requiring no additional data, fine-tuning, or inference-time overhead while largely preserving general multimodal capability.

Problem

• vision-language models
• object hallucination
• multimodal hallucination
• attention mechanisms
• model reliability

Innovation

• QK Product Steering
• training-free
• hallucination mitigation
• spectral decomposition
• grouped-query attention