🤖 AI Summary
This study presents the first systematic evaluation of mainstream conversational agents (e.g., Gemini, Grok) on cross-modal architecture design—specifically, their ability to autonomously generate an audio encoder that replaces the CLIP text encoder in Stable Diffusion 1.5 for end-to-end audio-to-image generation. Leveraging a dataset of over two million audio–image–text triplets, we prompt five large language models to design audio encoder architectures, then uniformly train and evaluate all generated models. Results show that while most models produce syntactically valid architectures, their audio embeddings fail to align with the original CLIP text embedding space. Gemini achieves the best quantitative performance (lowest FID, highest CLIP-Score), whereas Grok yields the most visually coherent generated images. Our work exposes fundamental structural limitations and consistent misalignment biases in current LLMs for cross-modal representation alignment tasks, establishing an empirical benchmark and methodological insights for AI-driven neural architecture search.
📝 Abstract
On one hand, recent advances in chatbots has led to a rising popularity in using these models for coding tasks. On the other hand, modern generative image models primarily rely on text encoders to translate semantic concepts into visual representations, even when there is clear evidence that audio can be employed as input as well. Given the previous, in this work, we explore whether state-of-the-art conversational agents can design effective audio encoders to replace the CLIP text encoder from Stable Diffusion 1.5, enabling image synthesis directly from sound. We prompted five publicly available chatbots to propose neural architectures to work as these audio encoders, with a set of well-explained shared conditions. Each valid suggested encoder was trained on over two million context related audio-image-text observations, and evaluated on held-out validation and test sets using various metrics, together with a qualitative analysis of their generated images. Although almost all chatbots generated valid model designs, none achieved satisfactory results, indicating that their audio embeddings failed to align reliably with those of the original text encoder. Among the proposals, the Gemini audio encoder showed the best quantitative metrics, while the Grok audio encoder produced more coherent images (particularly, when paired with the text encoder). Our findings reveal a shared architectural bias across chatbots and underscore the remaining coding gap that needs to be bridged in future versions of these models. We also created a public demo so everyone could study and try out these audio encoders. Finally, we propose research questions that should be tackled in the future, and encourage other researchers to perform more focused and highly specialized tasks like this one, so the respective chatbots cannot make use of well-known solutions and their creativity/reasoning is fully tested.