Parse Tree Tracking Through Time for Programming Process Analysis at Scale

2025-09-03

AI Summary

This paper addresses the challenge of automatically tracking abstract syntax tree (AST) nodes across time—particularly during incremental code editing when source code is temporarily unparsable. We propose a novel algorithm enabling cross-timestamp AST node tracking by integrating keystroke-level edit logging with syntactic recovery for incomplete code, thereby enabling structured AST construction and continuous evolution even for non-well-formed programs. Our contributions are threefold: (1) fine-grained behavioral analysis of core syntactic constructs (e.g., conditionals and loops); (2) scalable quantification of programming processes, revealing previously undocumented patterns—such as ~33% of commented-out code being subsequently restored, and comparable deletion rates inside versus outside syntactic structures; and (3) an extensible foundational infrastructure for cognitive studies of programming and intelligent educational diagnostics.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Reasoning under Uncertainty: Probabilistic Programming
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Background and Context: Programming process data can be utilized to understand the processes students use to write computer programming assignments. Keystroke- and line-level event logs have been used in the past in various ways, primarily in high-level descriptive statistics (e.g., timings, character deletion rate, etc). Analysis of behavior in context (e.g., how much time students spend working on loops) has been cumbersome because of our inability to automatically track high-level code representations, such as abstract syntax trees, through time and unparseable states. Objective: Our study has two goals. The first is to design the first algorithm that tracks parse tree nodes through time. Second, we utilize this algorithm to perform a partial replication study of prior work that used manual tracking of code representations, as well as other novel analyses of student programming behavior that can now be done at scale. Method: We use two algorithms presented in this paper to track parse tree nodes through time and construct tree representations for unparseable code states. We apply these algorithms to a public keystroke data from student coursework in a 2021 CS1 course and conduct analysis on the resulting parse trees. Findings: We discover newly observable statistics at scale, including that code is deleted at similar rates inside and outside of conditionals and loops, a third of commented out code is eventually restored, and that frequency with which students jump around in their code may not be indicative of struggle. Implications: The ability to track parse trees through time opens the door to understanding new dimensions of student programming, such as best practices of structural development of code over time, quantitative measurement of what syntactic constructs students struggle most with, refactoring behavior, and attention shifting within the code.

Problem

• Tracking parse tree nodes through time automatically
• Analyzing student programming behavior at scale
• Handling unparseable code states in process analysis

Innovation

• Algorithm tracks parse tree nodes through time
• Constructs tree representations for unparseable states
• Enables programming behavior analysis at scale