Improving Physical Object State Representation in Text-to-Image Generative Systems

2025-05-04

AI Summary

Current text-to-image models struggle to accurately model physical object states—such as “a table without a bottle” or “an empty cup”—particularly failing to comprehend abstract states involving negation or emptiness. To address this, we propose the first fully automated synthetic data generation method specifically designed for object state modeling, producing high-quality, fine-grained state annotations. We introduce the first dedicated evaluation benchmark: a 200-prompt, fine-grained prompt set covering diverse everyday object states. Leveraging LoRA-based fine-tuning on open-source diffusion models and employing GPT-4o-mini as a vision-language alignment evaluator, we establish a dual-track evaluation framework integrating GenAI-Bench and our custom state-centric benchmark. Experiments demonstrate an average absolute improvement of over 8% on GenAI-Bench and over 24% on our custom prompt set. All generated data, source code, and evaluation protocols are publicly released.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Current text-to-image generative models struggle to accurately represent object states (e.g.,"a table without a bottle,""an empty tumbler"). In this work, we first design a fully-automatic pipeline to generate high-quality synthetic data that accurately captures objects in varied states. Next, we fine-tune several open-source text-to-image models on this synthetic data. We evaluate the performance of the fine-tuned models by quantifying the alignment of the generated images to their prompts using GPT4o-mini, and achieve an average absolute improvement of 8+% across four models on the public GenAI-Bench dataset. We also curate a collection of 200 prompts with a specific focus on common objects in various physical states. We demonstrate a significant improvement of an average of 24+% over the baseline on this dataset. We release all evaluation prompts and code.

Problem

• Enhancing object state accuracy in text-to-image models
• Generating synthetic data for varied object states
• Improving image-prompt alignment for physical states

Innovation

• Automated pipeline for synthetic data generation
• Fine-tuning models with synthetic data
• Improved alignment using GPT4o-mini evaluation