Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation

2026-04-14

AI Summary

This work addresses the susceptibility of multimodal large language models to hallucinations during reasoning, often caused by linguistic priors overwhelming visual evidence. The authors propose a training-free, decoding-stage intervention framework that attributes such hallucinations to the model’s hypersensitivity to textual phrasing. By dynamically perturbing input text to probe the direction of prior drift, the method corrects probability biases through a combination of multi-level perturbations, attention variance analysis, and logits statistics. This approach constructs dynamic probes and a feature-space noise suppression mechanism, significantly reducing hallucination rates across multiple benchmarks while preserving generation fluency. The proposed technique outperforms existing methods without requiring any model retraining or fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Large Language Models frequently suffer from inference hallucinations, partially stemming from language priors dominating visual evidence. Existing training-free mitigation methods either perturb the visual representation and deviate from the natural image distribution, or enforce intrusive manipulations that compromise the model's inherent generative fluency. We introduce a novel perspective that multimodal hallucination manifests as the hypersensitivity of visual grounding to textual phrasing during the decoding phase. Building on this insight, we propose Decoding by Perturbation (DeP), a training-free framework mitigating prior-induced hallucinations via controlled textual interventions. DeP employs a dynamic probe applying multi-level textual perturbations to elicit latent language priors. Leveraging attention variance, it enhances stable evidence regions while suppressing suspicious noise in the feature space. Furthermore, it constructs an interpretable prior drift direction using logits statistics to counteract probability biases from textual co-occurrences. Extensive experiments confirm DeP effectively reduces hallucinations and achieves superior performance across multiple benchmarks.

Problem

• multimodal hallucination
• language priors
• visual grounding
• decoding phase
• prior-induced hallucinations

Innovation

• Decoding by Perturbation
• multimodal hallucination
• textual perturbation
• language priors
• attention variance