TalkPhoto: A Versatile Training-Free Conversational Assistant for Intelligent Image Editing

2026-01-05

arXiv.org

AI Summary

Existing instruction-based image editing methods rely heavily on large-scale, multi-turn instruction data for training, limiting their ability to efficiently handle diverse and complex editing requests. This work proposes the first training-free, general-purpose image editing framework that interprets user intent through conversational interaction by leveraging open-source large language models and carefully designed prompt templates to hierarchically orchestrate state-of-the-art editing tools. The approach employs a modular tool integration mechanism that enables plug-and-play extension of new editing capabilities. Experiments demonstrate that this method significantly improves both editing quality and tool selection accuracy across a wide range of tasks while reducing token consumption, consistently outperforming existing training-dependent approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Thanks to the powerful language comprehension capabilities of Large Language Models (LLMs), existing instruction-based image editing methods have introduced Multimodal Large Language Models (MLLMs) to promote information exchange between instructions and images, ensuring the controllability and flexibility of image editing. However, these frameworks often build a multi-instruction dataset to train the model to handle multiple editing tasks, which is not only time-consuming and labor-intensive but also fails to achieve satisfactory results. In this paper, we present TalkPhoto, a versatile training-free image editing framework that facilitates precise image manipulation through conversational interaction. We instruct the open-source LLM with a specially designed prompt template to analyze user needs after receiving instructions and hierarchically invoke existing advanced editing methods, all without additional training. Moreover, we implement a plug-and-play and efficient invocation of image editing methods, allowing complex and unseen editing tasks to be integrated into the current framework, achieving stable and high-quality editing results. Extensive experiments demonstrate that our method not only provides more accurate invocation with fewer token consumption but also achieves higher editing quality across various image editing tasks.

Problem

• instruction-based image editing
• multimodal large language models
• training overhead
• editing controllability
• dataset construction

Innovation

• training-free
• conversational image editing
• prompt engineering
• multimodal LLM
• plug-and-play editing