Can We Perform Online RL for Image Editing without Editing Rewards?

2026-08-24

AI Summary

本文探讨了在没有编辑奖励的情况下，通过将图像编辑维度映射到文本到图像生成的奖励空间，并引入Lever-Edit框架来解决在线RL图像编辑问题。

Abstract

Reinforcement learning (RL) enables direct preference optimization for image editing through editing-specific rewards, which remain less developed due to costly triplet supervision and complex task-dependent calibration. In contrast, text-to-image (T2I) generation benefits from a mature and diverse reward ecosystem spanning semantic alignment, aesthetics, realism, glyph shape, and other visual preferences. Extending this ecosystem to image editing would substantially broaden the range of visual preferences accessible to RL-based optimization, prompting the central question: \emph{Can We Perform Image Editing RL without Editing Rewards?} In this paper, we argue that the standard image editing dimensions have potential to be mapped to the T2I reward space: image quality can transfer directly, prompt following can be aligned through a description of the desired visual state, and reference consistency admits a coarse semantic conversion by encoding the source content to preserve. However, editing instructions specify relative changes, whereas T2I rewards require self-contained target descriptions; moreover, semantically valid captions from generic vision-language models may be incompatible with the frozen reward. Hence, we further introduce Lever-Edit, a two-stage framework that learns a reward-aligned captioner for counterfactual target descriptions, freezes it, and optimizes the editing policy solely with the transferred T2I reward. Experiments show competitive editing alignment and source preservation against editing-reward-based fine-tuning, while outperforming intuitive transfer baselines.

Problem

• Reinforcement Learning
• Image Editing
• Text-to-Image Generation
• Reward Alignment
• Editing Rewards

Innovation

• Lever-Edit
• Reward-aligned Captioner
• T2I Reward Ecosystem
• Image Editing RL