PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

2026-04-14

AI Summary

Existing text-to-image generation models lack high-quality, unlabeled reward signals for reinforcement learning: CLIP Score offers only coarse-grained feedback, while vision-language model (VLM)-based approaches rely on human preference data and additional fine-tuning. This work proposes PromptEcho, the first reward mechanism for image-text alignment that requires neither annotations nor reward model training. It leverages a frozen VLM to compute token-level cross-entropy loss between the generated image and the original prompt, directly utilizing the VLM’s pretrained knowledge as a fine-grained reward signal. The method automatically benefits from advances in open-source VLMs and achieves substantial gains on the newly introduced complex-prompt benchmark DenseAlignBench, improving win rates by 26.8 and 16.2 percentage points for Z-Image and QwenImage-2512, respectively, with consistent improvements also observed on GenEval, DPG-Bench, and TIIFBench.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reinforcement learning (RL) can improve the prompt following capability of text-to-image (T2I) models, yet obtaining high-quality reward signals remains challenging: CLIP Score is too coarse-grained, while VLM-based reward models (e.g., RewardDance) require costly human-annotated preference data and additional fine-tuning. We propose PromptEcho, a reward construction method that requires \emph{no} annotation and \emph{no} reward model training. Given a generated image and a guiding query, PromptEcho computes the token-level cross-entropy loss of a frozen VLM with the original prompt as the label, directly extracting the image-text alignment knowledge encoded during VLM pretraining. The reward is deterministic, computationally efficient, and improves automatically as stronger open-source VLMs become available. For evaluation, we develop DenseAlignBench, a benchmark of concept-rich dense captions for rigorously testing prompt following capability. Experimental results on two state-of-the-art T2I models (Z-Image and QwenImage-2512) demonstrate that PromptEcho achieves substantial improvements on DenseAlignBench (+26.8pp / +16.2pp net win rate), along with consistent gains on GenEval, DPG-Bench, and TIIFBench without any task-specific training. Ablation studies confirm that PromptEcho comprehensively outperforms inference-based scoring with the same VLM, and that reward quality scales with VLM size. We will open-source the trained models and the DenseAlignBench.

Problem

• text-to-image
• reinforcement learning
• reward signal
• vision-language models
• prompt following

Innovation

• annotation-free reward
• vision-language model
• text-to-image reinforcement learning
• token-level cross-entropy
• frozen VLM