Visual Puns from Idioms: An Iterative LLM-T2IM-MLLM Framework

2025-11-28

AI Summary

This work addresses the task of automatically generating idiomatic visual pun images—images that simultaneously convey both the literal and figurative meanings of an idiom. We propose the first multimodal iterative closed-loop framework for this task: a large language model (LLM) generates visual prompts; a text-to-image (T2I) model synthesizes candidate images; and a multimodal large language model (MLLM) performs reverse idiom recognition to identify which idiom the generated image implicitly represents, enabling prompt refinement via feedback. To support systematic evaluation of both generation quality and idiom recognition capability, we introduce the first benchmark dataset comprising 1,000 idioms, along with corresponding ground-truth images and prompts. Experimental results demonstrate that MLLM selection critically impacts performance: GPT-4 achieves the best overall results, followed by Gemini; open-source models such as Gemma show competitive capability, while Claude excels specifically in prompt generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We study idiom-based visual puns--images that align an idiom's literal and figurative meanings--and present an iterative framework that coordinates a large language model (LLM), a text-to-image model (T2IM), and a multimodal LLM (MLLM) for automatic generation and evaluation. Given an idiom, the system iteratively (i) generates detailed visual prompts, (ii) synthesizes an image, (iii) infers the idiom from the image, and (iv) refines the prompt until recognition succeeds or a step limit is reached. Using 1,000 idioms as inputs, we synthesize a corresponding dataset of visual pun images with paired prompts, enabling benchmarking of both generation and understanding. Experiments across 10 LLMs, 10 MLLMs, and one T2IM (Qwen-Image) show that MLLM choice is the primary performance driver: GPT achieves the highest accuracies, Gemini follows, and the best open-source MLLM (Gemma) is competitive with some closed models. On the LLM side, Claude attains the strongest average performance for prompt generation.

Problem

• Generates visual pun images from idioms automatically
• Evaluates idiom recognition in images using multimodal models
• Benchmarks performance of various LLMs and MLLMs for generation

Innovation

• Iterative LLM-T2IM-MLLM framework for visual puns
• Generates and refines prompts until idiom recognition succeeds
• Synthesizes dataset for benchmarking generation and understanding