Controllable Image Captioning with Prompt-Conditioned Scene Rewards

2026-09-01

AI Summary

研究解决了图像描述生成中语义控制不足的问题，通过基于场景奖励和自然语言提示的方法，增强了对描述内容的细粒度控制。

Abstract

Large Vision-Language Models produce fluent image descriptions but offer limited semantic control: users cannot reliably specify whether captions should emphasize attributes, relations, or particular image regions. We present Fine-grained Captioning Control Using Scene Rewards (FoCUS), a controllable image captioning method that lets users steer captions toward specific semantic emphases through natural-language control prompts. The core idea is a prompt-conditioned control objective based on scene-graph-aligned component scores. Generated captions are parsed and aligned to scene-graph components such as objects, attributes, and relations. These components are differentially weighted, including negative weights, according to the requested emphasis. We optimize this objective with GRPO and further improve its reliability through a stricter object validity threshold and reasoning-based verification for attribute and relation scoring. To evaluate controllability, we introduce Semantic Control and Precision Evaluation (SCoPE), a benchmark with contrastive Include/Avoid constraints for measuring both target content coverage and out-of-scope suppression. Experiments on two VLM backbones show that FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.

Problem

• Controllable Image Captioning
• Semantic Control
• Vision-Language Models

Innovation

• Controllable Image Captioning
• Scene Graphs
• Natural-Language Prompts
• Component Scores
• GRPO Optimization