Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering

2026-05-28

AI Summary

This work addresses the tendency of large vision-language models to generate hallucinations during late-stage decoding due to degraded visual grounding. Existing inference-time intervention methods suffer from the absence of explicit objectives, fixed correction strength, and excessive interference. To overcome these limitations, we propose BRACS, a novel framework that introduces, for the first time, a barrier-constrained adaptive closed-form guidance mechanism. BRACS dynamically monitors the model’s internal attention to assess grounding quality and applies on-demand, variable-strength analytical corrections to hidden states only when grounding deteriorates—without requiring additional training. Experiments show that BRACS reduces CHAIR$_s$ by 9.4 points and improves POPE F1 by 2.7 points on LLaVA-1.5-7B and Qwen-VL-Chat, while matching or surpassing baseline performance across four general multimodal benchmarks. The method achieves 80% of greedy decoding speed, yielding an average 1.3× acceleration.

Abstract

Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIR$_s$ by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.

Problem

• hallucination
• vision-language models
• visual grounding
• multimodal generation

Innovation

• hallucination mitigation
• vision-language models
• adaptive steering
• closed-form correction
• visual grounding