EmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation

2026-07-04

AI Summary

Existing approaches struggle to directly assess the extent to which emerging languages encode the visual content of their input images—referred to as visual reflectivity. This work proposes the first image-generation-based framework for direct evaluation: by fine-tuning a pretrained text-to-image diffusion model to reconstruct the original image from messages in the emerging language, and quantifying visual reflectivity through the perceptual similarity between reconstructed and original images. The method obviates the need for human-defined concepts or proxy metrics, thereby capturing visual information in language more comprehensively. Experiments on MS-COCO demonstrate that the approach significantly outperforms random and fixed-token baselines and reveals discrepancies in visual content that four existing evaluation metrics fail to capture.

Abstract

Measuring the extent to which emergent languages encode the visual content of their inputs is an open problem. We refer to this property as visual reflection: the extent to which emergent messages preserve information about their source images that can be recovered without appeal to the speaker-listener pair that produced them. Existing metrics measure it only indirectly, through proxies such as human-defined concept inventories, natural-language captions, structural distance correlations, or Referential Game accuracy, each of which can either miss visual content the message encodes or credit content it does not. We propose EmCom-Diffusion, an evaluation framework that measures visual reflection directly: it reconstructs each input image from its emergent message and compares the reconstruction with the original image itself, rather than with human-defined targets. Concretely, it finetunes a pretrained text-to-image diffusion model on (image, emergent-message) pairs and scores visual reflection as the perceptual similarity between the reconstructed and original images, operating generatively rather than discriminatively. Instantiating it on MS-COCO with a Referential Game, we validate the metric against random and fixed-token baselines under three pretrained visual encoders, and compare it against four existing metrics (CBM, supervised translation, TopSim, and R@1). EmCom-Diffusion captures visual content the other metrics miss.

Problem

• emergent languages
• visual reflection
• image reconstruction
• evaluation metric
• visual content encoding

Innovation

• emergent languages
• visual reflection
• diffusion models
• image reconstruction
• perceptual similarity