InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs

2023-12-15

AI Summary

High entry barriers for beginners learning machine learning (ML) and the substantial development cost of visual programming environments hinder widespread adoption. To address these challenges, this paper proposes VisuPipe—a low-code, visualization-oriented ML pipeline generation framework powered by dual large language models (LLMs). Its core innovation lies in a novel “pseudocode generation + executable node-graph rendering” two-module paradigm, enabling natural-language-driven, AI-human collaborative pipeline construction and iterative refinement. VisuPipe tightly integrates an LLM, a code interpreter, and dynamic visualization rendering to achieve end-to-end translation from textual specifications to executable, interactive node graphs. A user study (N=16) demonstrates that VisuPipe significantly reduces the initial learning curve—cutting average modeling time by 62%—enhances pipeline construction efficiency, and fosters innovative modeling approaches.

Abstract

Visual programming has the potential of providing novice programmers with a low-code experience to build customized processing pipelines. Existing systems typically require users to build pipelines from scratch, implying that novice users are expected to set up and link appropriate nodes from a blank workspace. In this paper, we introduce InstructPipe, an AI assistant for prototyping machine learning (ML) pipelines with text instructions. We contribute two large language model (LLM) modules and a code interpreter as part of our framework. The LLM modules generate pseudocode for a target pipeline, and the interpreter renders the pipeline in the node-graph editor for further human-AI collaboration. Both technical and user evaluation (N=16) shows that InstructPipe empowers users to streamline their ML pipeline workflow, reduce their learning curve, and leverage open-ended commands to spark innovative ideas.

Problem

• Simplifying ML pipeline creation for novice programmers
• Reducing learning curve with AI-generated pseudocode
• Enhancing human-AI collaboration in visual programming

Innovation

• LLM modules generate pseudocode
• Code interpreter renders pipelines
• Streamlines ML workflow with text instructions