CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMs

2025-10-13

AI Summary

How can fine-grained, real-time interaction data between developers and code generation tools (CGTs) be collected without disrupting their workflow? This paper proposes a lightweight, non-intrusive client–server system: a VS Code extension captures semantically rich editing events—including code insertion/deletion, copy-paste operations, and focus transitions—while a Python-based RESTful API persistently stores structured telemetry in MongoDB. All components are containerized to ensure scalability and deployment simplicity. The key contribution is the first implementation enabling seamless, session-level behavioral reconstruction and analysis—achieving high-fidelity, low-overhead continuous monitoring of developer–CGT interactions. This infrastructure supports rigorous behavioral modeling, productivity assessment, and responsible AI research in AI-augmented programming environments.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Interaction Techniques and Devices

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Understanding how developers interact with code generation tools (CGTs) requires detailed, real-time data on programming behavior which is often difficult to collect without disrupting workflow. We present extit{CodeWatcher}, a lightweight, unobtrusive client-server system designed to capture fine-grained interaction events from within the Visual Studio Code (VS Code) editor. extit{CodeWatcher} logs semantically meaningful events such as insertions made by CGTs, deletions, copy-paste actions, and focus shifts, enabling continuous monitoring of developer activity without modifying user workflows. The system comprises a VS Code plugin, a Python-based RESTful API, and a MongoDB backend, all containerized for scalability and ease of deployment. By structuring and timestamping each event, extit{CodeWatcher} enables post-hoc reconstruction of coding sessions and facilitates rich behavioral analyses, including how and when CGTs are used during development. This infrastructure is crucial for supporting research on responsible AI, developer productivity, and the human-centered evaluation of CGTs. Please find the demo, diagrams, and tool here: https://osf.io/j2kru/overview.

Problem

• Captures fine-grained developer interactions with code generation tools
• Enables continuous monitoring of coding behavior without workflow disruption
• Facilitates post-hoc reconstruction of coding sessions for behavioral analysis

Innovation

• Lightweight client-server system for IDE telemetry
• Logs fine-grained coding interactions without workflow disruption
• Containerized architecture enabling scalable behavioral analysis