Controlling Multimodal LLMs via Reward-guided Decoding

2025-08-15

AI Summary

Weak visual grounding and poor generation controllability hinder multimodal large language models (MLLMs). To address these issues, this paper proposes Reward-Guided Decoding (RGD), a novel inference-time framework. RGD introduces the first dual reward modeling scheme—separately quantifying localization precision and recall—and dynamically coordinates them during autoregressive decoding. It integrates lightweight reward modeling, controllable sampling strategies, and adaptive test-time computational resource allocation to enable fine-grained, real-time generation intervention. Evaluated on standard hallucination benchmarks, RGD significantly outperforms existing hallucination-mitigation methods: it preserves linguistic coherence while substantially improving the balance between visual object localization accuracy and recall. This work establishes a new paradigm for controllable multimodal generation in MLLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As Multimodal Large Language Models (MLLMs) gain widespread applicability, it is becoming increasingly desirable to adapt them for diverse user needs. In this paper, we study the adaptation of MLLMs through controlled decoding. To achieve this, we introduce the first method for reward-guided decoding of MLLMs and demonstrate its application in improving their visual grounding. Our method involves building reward models for visual grounding and using them to guide the MLLM's decoding process. Concretely, we build two separate reward models to independently control the degree of object precision and recall in the model's output. Our approach enables on-the-fly controllability of an MLLM's inference process in two ways: first, by giving control over the relative importance of each reward function during decoding, allowing a user to dynamically trade off object precision for recall in image captioning tasks; second, by giving control over the breadth of the search during decoding, allowing the user to control the trade-off between the amount of test-time compute and the degree of visual grounding. We evaluate our method on standard object hallucination benchmarks, showing that it provides significant controllability over MLLM inference, while consistently outperforming existing hallucination mitigation methods.

Problem

• Adapting MLLMs for diverse user needs via controlled decoding
• Improving visual grounding in MLLMs using reward-guided decoding
• Controlling object precision and recall trade-offs in MLLM outputs

Innovation

• Reward-guided decoding for MLLM control
• Separate reward models for precision and recall
• Dynamic trade-off control during decoding