Drawing Your Programs: Exploring the Applications of Visual-Prompting with GenAI for Teaching and Assessment

2026-02-11

AI Summary

This work addresses the overreliance on textual prompts in current human-AI collaborative programming, which overlooks the visual and spatial representations naturally employed by programmers during design. It systematically proposes and empirically validates, for the first time, that hand-drawn problem decomposition diagrams produced by students can serve as effective visual prompts to directly guide multimodal generative AI models—such as GPT-4o—in producing correct code. Drawing on large-scale data from an introductory Python programming course, the study demonstrates that this approach significantly extends the application paradigm of generative AI in programming education and assessment, offering preliminary evidence for the efficacy of multimodal prompting in computing education.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

When designing a program, both novice programmers and seasoned developers alike often sketch out -- or, perhaps more famously, whiteboard -- their ideas. Yet despite the introduction of natively multimodal Generative AI models, work on Human-GenAI collaborative coding has remained overwhelmingly focused on textual prompts -- largely ignoring the visual and spatial representations that programmers naturally use to reason about and communicate their designs. In this proposal and position paper, we argue and provide tentative evidence that this text-centric focus overlooks other forms of prompting GenAI models, such as problem decomposition diagrams functioning as prompts for code generation in their own right enabling new types of programming activities and assessments. To support this position, we present findings from a large introductory Python programming course, where students constructed decomposition diagrams that were used to prompt GPT-4.1 for code generation. We demonstrate that current models are very successful in their ability to generate code from student-constructed diagrams. We conclude by exploring the implications of embracing multimodal prompting for computing education, particularly in the context of assessment.

Problem

• visual prompting
• Generative AI
• programming education
• multimodal interaction
• code generation

Innovation

• visual prompting
• multimodal GenAI
• program decomposition diagrams
• code generation
• computing education