PointT2I: LLM-based text-to-image generation via keypoints

2025-06-02

AI Summary

To address pose misalignment and poor text–image alignment in text-to-image (T2I) generation for complex human pose descriptions, this paper proposes the first fine-tuning-free, purely text-driven keypoint-guided generation framework. Methodologically: (1) it leverages large language models (LLMs) to directly parse input text and generate structured human keypoint sequences; (2) it introduces an LLM-driven semantic consistency feedback mechanism that dynamically verifies and refines the generation process; and (3) it designs a multimodal conditional image generation module that jointly conditions on both text and keypoints for end-to-end pose alignment. Crucially, the framework requires no external pose annotations or model fine-tuning. Experiments demonstrate significant improvements in pose accuracy and image fidelity while preserving strong text fidelity—achieving state-of-the-art alignment without compromising textual grounding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Text-to-image (T2I) generation model has made significant advancements, resulting in high-quality images aligned with an input prompt. However, despite T2I generation's ability to generate fine-grained images, it still faces challenges in accurately generating images when the input prompt contains complex concepts, especially human pose. In this paper, we propose PointT2I, a framework that effectively generates images that accurately correspond to the human pose described in the prompt by using a large language model (LLM). PointT2I consists of three components: Keypoint generation, Image generation, and Feedback system. The keypoint generation uses an LLM to directly generate keypoints corresponding to a human pose, solely based on the input prompt, without external references. Subsequently, the image generation produces images based on both the text prompt and the generated keypoints to accurately reflect the target pose. To refine the outputs of the preceding stages, we incorporate an LLM-based feedback system that assesses the semantic consistency between the generated contents and the given prompts. Our framework is the first approach to leveraging LLM for keypoints-guided image generation without any fine-tuning, producing accurate pose-aligned images based solely on textual prompts.

Problem

• Generating images from complex human pose descriptions
• Accurately aligning images with text prompts using keypoints
• Leveraging LLMs for keypoint-guided image generation without fine-tuning

Innovation

• LLM generates keypoints from text prompts
• Keypoints guide image generation for accuracy
• LLM feedback ensures semantic consistency