How Do Inpainting Artifacts Propagate to Language?

2026-02-24

AI Summary

This study investigates how visual artifacts introduced by diffusion-based image inpainting affect the language generation quality of vision-language models. The authors propose a two-stage diagnostic framework: first inpainting masked image regions and then feeding both original and inpainted images into an image captioning model to compare their generated descriptions. This work establishes, for the first time, a quantitative relationship between visual reconstruction fidelity and language generation performance, revealing that inpainting artifacts induce hierarchical shifts in the internal representations and attention mechanisms of multimodal models. Through comprehensive evaluation combining pixel-level and perceptual metrics, intermediate visual feature analysis, and attention visualization, the study demonstrates that reconstruction fidelity significantly influences both word-level accuracy and semantic coherence in generated captions.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then provided to captioning models, enabling controlled comparisons between captions generated from original and reconstructed inputs. Across multiple datasets, we analyze the relationship between reconstruction fidelity and downstream caption quality. We observe consistent associations between pixel-level and perceptual reconstruction metrics and both lexical and semantic captioning performance. Additional analysis of intermediate visual representations and attention patterns shows that inpainting artifacts lead to systematic, layer-dependent changes in model behavior. Together, these results provide a practical diagnostic framework for examining how visual reconstruction quality influences language generation in multimodal systems.

Problem

• inpainting artifacts
• vision-language models
• language generation
• reconstruction fidelity
• multimodal systems

Innovation

• inpainting artifacts
• vision-language models
• diffusion-based reconstruction
• multimodal diagnostics
• caption generation