Text2VP: Generative AI for Visual Programming and Parametric Modeling

2024-06-09

arXiv.org

AI Summary

Existing generative AI systems lack direct support for translating architectural design specifications from natural language into parametric models, hindering rapid generation and optimization of diverse design alternatives. Method: We propose Text-to-Visual Programming (Text2VP) GPT—a novel framework enabling end-to-end generation of executable visual programming scripts (e.g., for Grasshopper) from natural language. Our approach introduces a vision-programming–oriented text generation paradigm integrating API documentation parsing, structured prompt engineering, visual syntax–constrained decoding, and interactive parameter validation. It supports instruction-driven, documentation-augmented, and example-guided few-shot inference to jointly generate parametric logic, node connectivity, and data flow. Contribution/Results: Evaluated on a standardized benchmark, Text2VP achieves 82.3% functional correctness. It reliably produces runnable, medium-to-low-complexity parametric models and enables real-time interactive refinement—significantly lowering the barrier to entry for domain-specific visual programming tools.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The integration of generative artificial intelligence (AI) into architectural design has advanced significantly, enabling the generation of text, images, and 3D models. However, prior AI applications lack support for text-to-parametric models, essential for generating and optimizing diverse parametric design options. This study introduces Text-to-Visual Programming (Text2VP) GPT, a novel generative AI derived from GPT-4.1, designed to automate graph-based visual programming workflows, parameters, and their interconnections. Text2VP leverages detailed documentation, specific instructions, and example-driven few-shot learning to reflect user intentions accurately and facilitate interactive parameter adjustments. Testing demonstrates Text2VP's capability in generating functional parametric models, although higher complexity models present increased error rates. This research highlights generative AI's potential in visual programming and parametric modeling, laying groundwork for future improvements to manage complex modeling tasks. Ultimately, Text2VP aims to enable designers to easily create and modify parametric models without extensive training in specialized platforms like Grasshopper.

Problem

• Lack of AI support for text-to-parametric model conversion
• Automating visual programming workflows and parameter connections
• Enabling parametric design without extensive platform training

Innovation

• Text-to-Visual Programming (Text2VP) GPT automates workflows
• Uses few-shot learning for accurate user intentions
• Generates functional parametric models from text