Sketch Bug: Using Sketch-Based Input for Interactive Code Debugging

2026-05-22

AI Summary

This work proposes a sketch-based handwritten input approach to interactive debugging, addressing the limitations of traditional tools that rely on discrete mouse-and-keyboard operations and lack support for spatially continuous interaction. By integrating gesture recognition with Python execution tracing, the method enables programmers to set breakpoints through lightweight annotations, control execution flow using symbolic strokes, and navigate loops via spiral gestures. Implemented within a standard code editor, the approach was evaluated in a controlled study with 24 developers, demonstrating its effectiveness in breakpoint placement, stepwise execution, and state inspection. The findings also highlight challenges related to gesture recognition accuracy and cognitive load, thereby expanding the boundaries of human-computer interaction in program debugging.

Abstract

We investigate sketch-like pen input as an alternative way to support execution control in interactive debugging. In our interface, programmers draw lightweight marks to set breakpoints, use symbolic strokes to control execution, and extend strokes into spirals to repeat traversal actions. The prototype combines gesture recognition with Python execution tracing in a conventional editor interface. In a controlled study with 24 programmers, we compared the sketch interface with conventional mouse-and-keyboard input on debugging tasks that required breakpoint placement, step-wise execution, and runtime state inspection. The results show that sketch-like input can support these execution-control tasks, while also introducing challenges in precision, recognition, and gesture recall. Our findings suggest that pen input is most promising where debugger interactions benefit from spatial grounding or continuous movement, rather than as a wholesale replacement for conventional debugging controls.

Problem

• sketch-based input
• interactive debugging
• execution control
• gesture recognition
• code debugging

Innovation

• sketch-based interaction
• gesture recognition
• interactive debugging
• pen input
• execution control