3Dify: a Framework for Procedural 3D-CG Generation Assisted by LLMs Using MCP and RAG

2025-10-06

AI Summary

To address weak interactivity, low generation quality, and heavy reliance on external APIs in natural language–driven 3D-CG generation, this paper proposes an end-to-end framework built on the Dify platform. Methodologically, it innovatively integrates the Model Context Protocol (MCP) with Computer Usage Agents (CUA) to enable generic GUI automation across multiple DCC tools; it further combines Retrieval-Augmented Generation (RAG) with a locally deployed lightweight LLM to support user-feedback-driven iterative refinement and personalized model customization. Key contributions include: (1) the first application of MCP-CUA synergy to the 3D content creation pipeline, significantly improving instruction understanding accuracy and geometric/texture generation quality; (2) reduction of external API calls by over 90%, ~65% decrease in per-generation latency; and (3) full offline capability and low-cost deployment.

Technology Category

• Natural Language Processing: Generation
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Computer Vision: 3D Computer Vision

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This paper proposes "3Dify," a procedural 3D computer graphics (3D-CG) generation framework utilizing Large Language Models (LLMs). The framework enables users to generate 3D-CG content solely through natural language instructions. 3Dify is built upon Dify, an open-source platform for AI application development, and incorporates several state-of-the-art LLM-related technologies such as the Model Context Protocol (MCP) and Retrieval-Augmented Generation (RAG). For 3D-CG generation support, 3Dify automates the operation of various Digital Content Creation (DCC) tools via MCP. When DCC tools do not support MCP-based interaction, the framework employs the Computer-Using Agent (CUA) method to automate Graphical User Interface (GUI) operations. Moreover, to enhance image generation quality, 3Dify allows users to provide feedback by selecting preferred images from multiple candidates. The LLM then learns variable patterns from these selections and applies them to subsequent generations. Furthermore, 3Dify supports the integration of locally deployed LLMs, enabling users to utilize custom-developed models and to reduce both time and monetary costs associated with external API calls by leveraging their own computational resources.

Problem

• Generating 3D graphics through natural language instructions
• Automating DCC tool operations using MCP and GUI automation
• Enhancing image quality via user feedback and local LLMs

Innovation

• Automates DCC tools via MCP and CUA methods
• Learns user preferences from image selection feedback
• Integrates local LLMs to reduce API costs