Prompt-Guided Image Editing with Masked Logit Nudging in Visual Autoregressive Models

2026-04-15

AI Summary

This work proposes Masked Logit Nudging, a method for prompt-guided local image editing that strictly preserves non-edited regions. Built upon Vision Autoregressive (VAR) models, the approach generates spatial masks via cross-attention differences and applies semantic guidance to the predicted logits of target regions using token mappings from the source image, complemented by quantization error correction to enhance reconstruction fidelity. Evaluated on the PIE benchmark, the method achieves state-of-the-art editing performance at both 512px and 1024px resolutions, surpasses existing approaches in reconstruction quality on COCO and OpenImages datasets, and demonstrates significantly faster inference than diffusion-based models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We address the problem of prompt-guided image editing in visual autoregressive models. Given a source image and a target text prompt, we aim to modify the source image according to the target prompt, while preserving all regions which are unrelated to the requested edit. To this end, we present Masked Logit Nudging, which uses the source image token maps to introduce a guidance step that aligns the model's predictions under the target prompt with these source token maps. Specifically, we convert the fixed source encodings into logits using the VAR encoding, nudging the model's predicted logits towards the targets along a semantic trajectory defined by the source-target prompts. Edits are applied only within spatial masks obtained through a dedicated masking scheme that leverages cross-attention differences between the source and edited prompts. Then, we introduce a refinement to correct quantization errors and improve reconstruction quality. Our approach achieves the best image editing performance on the PIE benchmark at 512px and 1024px resolutions. Beyond editing, our method delivers faithful reconstructions and outperforms previous methods on COCO at 512px and OpenImages at 1024px. Overall, our method outperforms VAR-related approaches and achieves comparable or even better performance than diffusion models, while being much faster. Code is available at 'https://github.com/AmirMaEl/MLN'.

Problem

• prompt-guided image editing
• visual autoregressive models
• image editing
• text-to-image

Innovation

• Masked Logit Nudging
• Visual Autoregressive Models
• Prompt-Guided Image Editing
• Semantic Trajectory Alignment
• Cross-Attention Masking