Code Shaping: Iterative Code Editing with Free-form AI-Interpreted Sketching

2025-02-06

AI Summary

This paper addresses the inefficiency in intent communication between programmers and AI during collaborative code editing. We propose “code sculpting”—a paradigm enabling direct, freehand sketching over code and console output to guide AI in iterative code editing. Through a three-phase design study involving 18 professional programmers, we developed a sketch semantics model and a human-AI feedback mechanism, implemented an interactive prototype, and validated real-world use cases. Key contributions include: (1) the first systematic mapping between sketch types and editing intents; (2) empirical identification of programmers’ collaborative strategies for refining sketch semantics; (3) distilled design principles for tight integration between code editors and sketching interfaces; and (4) empirical evidence demonstrating significantly improved intent conveyance efficiency, alongside two deployable industrial applications.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: The sharing economy

Abstract

We introduce the concept of code shaping, an interaction paradigm for editing code using free-form sketch annotations directly on top of the code and console output. To evaluate this concept, we conducted a three-stage design study with 18 different programmers to investigate how sketches can communicate intended code edits to an AI model for interpretation and execution. The results show how different sketches are used, the strategies programmers employ during iterative interactions with AI interpretations, and interaction design principles that support the reconciliation between the code editor and sketches. Finally, we demonstrate the practical application of the code shaping concept with two use case scenarios, illustrating design implications from the study.

Problem

• Editing code via free-form sketching
• AI interpretation of sketch annotations
• Iterative interaction between programmers and AI

Innovation

• Free-form sketch code editing
• AI-interpreted sketching interaction
• Iterative code shaping process