Programmer Visual Attention During Context-Aware Code Summarization

2024-05-28

arXiv.org

AI Summary

This study investigates the visual attention mechanisms employed by programmers when generating context-aware code summaries and their relationship with summary quality. Using eye-tracking experiments, we analyzed gaze distributions and reading behaviors of XY Java developers performing realistic code summarization tasks. Our key contributions are: (1) Attention patterns systematically improve with programming experience; (2) A significant inverted-U relationship exists between source code reading volume and summary quality—moderate reading enhances quality (p < 0.01), whereas excessive reading degrades it; (3) Context from the same package proves more informative than call-graph-based context. Empirical results further show that experienced summarizers significantly reduce word-level reading volume and regression frequency (p < 0.03). Collectively, these findings provide cognitive foundations and empirical evidence for designing intelligent code summarization tools that align with human attentional constraints and expertise development.

Technology Category

• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Data Visualization & Summarization
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Abridged: Programmer attention represents the visual focus of programmers on parts of the source code in pursuit of programming tasks. We conducted an in-depth human study with XY Java programmers, where each programmer generated summaries for 40 methods from five large Java projects over five one-hour sessions. We used eye-tracking equipment to map the visual attention of programmers while they wrote the summaries. We also rate the quality of each summary. We found eye-gaze patterns and metrics that define common behaviors between programmer attention during context-aware code summarization. Specifically, we found that programmers need to read significantly (p<0.01) fewer words and make significantly fewer revisits to words (p extless0.03) as they summarize more methods during a session, while maintaining the quality of summaries. We also found that the amount of source code a participant looks at correlates with a higher quality summary, but this trend follows a bell-shaped curve, such that after a threshold reading more source code leads to a significant decrease (p<0.01) in the quality of summaries. We also gathered insight into the type of methods in the project that provide the most contextual information for code summarization based on programmer attention. Specifically, we observed that programmers spent a majority of their time looking at methods inside the same class as the target method to be summarized. Surprisingly, we found that programmers spent significantly less time looking at methods in the call graph of the target method. We discuss how our empirical observations may aid future studies towards modeling programmer attention and improving context-aware automatic source code summarization.

Problem

• Analyze programmer visual attention during code summarization
• Identify eye-gaze patterns affecting summary quality
• Determine contextual methods influencing summarization efficiency

Innovation

• Eye-tracking maps programmer visual attention
• Fewer words read improves summary efficiency
• Same-class methods provide most context