KOALA: a Configurable Tool for Collecting IDE Data When Solving Programming Tasks

2025-06-26

AI Summary

Existing programming behavior capture tools exhibit significant limitations in granularity control, environmental event logging, and configuration flexibility. This paper introduces KOALA, a highly configurable JetBrains IDE plugin designed to address these gaps. KOALA enables fine-grained capture of code snapshots, IDE interactions—including execution, debugging, keyboard shortcuts, window focus switches, and file navigation—feature usage telemetry, embedded micro-questionnaires, and task distribution. It represents the first systematic collection of previously underexplored behavioral dimensions, such as IDE shortcut usage and focus transitions. The tool supports multi-level granularity adjustment and modular feature toggling, and natively exports data in the ProgSnap2 standard format. Deployed across two university courses, KOALA successfully collected longitudinal programming process data from 28 students, demonstrating robustness and scalability. Empirical analysis yielded pedagogically meaningful insights, including patterns in code completion preferences and debugging strategies.

Technology Category

• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Web measurements
• Security and Privacy: Data transparency and provenance

Abstract

Collecting data of students solving programming tasks is incredibly valuable for researchers and educators. It allows verifying that the students correctly apply the features and concepts they are taught, or finding students' misconceptions. However, existing data collection tools have limitations, e.g., no control over the granularity of the collected code, not collecting the specific events of the programming environment used, and overall being hard to configure. To overcome these limitations, we propose KOALA, a convenient and highly configurable tool for collecting code snapshots and feature usage from students solving programming tasks in JetBrains IDEs. The plugin can be installed in IDEs and configured to provide the students with the necessary tasks, enable or disable certain IDE features like code completion, and run surveys. During problem solving, the plugin collects code snapshots at the configured granularity, all IDE actions like running and debugging, as well as some data not collected in prior works, like employed hotkeys and switching focus between files. The collected data is sent to the server that comes with the tool, where it is stored and can be converted to the standardized ProgSnap2 format. To showcase the tool, we collected data from 28 students solving tasks in two courses within the IDE, highlighting some insights from this data.

Problem

• Collecting granular IDE data from programming tasks
• Overcoming limitations of existing data collection tools
• Configurable tool for tracking student coding behaviors

Innovation

• Configurable IDE plugin for data collection
• Tracks code snapshots and IDE actions
• Converts data to ProgSnap2 format