The Influence of Code Comments on the Perceived Helpfulness of Stack Overflow Posts

2025-08-27

AI Summary

This study investigates how code comment format—specifically block comments, inline comments, and no comments—affects developers’ perceived usefulness of Stack Overflow answers. Method: We conducted an online controlled experiment simulating realistic developer browsing behavior, collecting both survey-based perceptions (e.g., helpfulness ratings) and behavioral data (e.g., dwell time, scrolling patterns). Contribution/Results: Results show that annotated code significantly enhances perceived answer helpfulness; block comments are rated as more helpful than inline comments—especially by novice developers—and this effect remains robust after controlling for confounding surface features such as answer position and score. This is the first empirical demonstration that comment structure critically influences knowledge dissemination efficiency in developer communities. The findings provide actionable design guidelines for improving AI programming assistants (e.g., prompting models to generate block comments by default) and enhancing readability and pedagogical suitability of open-source documentation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Crowd Sourcing and Human Computation
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Community question answering
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Question-and-answer platforms such as Stack Overflow have become an important way for software developers to share and retrieve knowledge. However, reusing poorly understood code can lead to serious problems, such as bugs or security vulnerabilities. To better understand how code comments affect the perceived helpfulness of Stack Overflow answers, we conducted an online experiment simulating a Stack Overflow environment (n=91). The results indicate that both block and inline comments are perceived as significantly more helpful than uncommented source code. Moreover, novices rated code snippets with block comments as more helpful than those with inline comments. Interestingly, other surface features, such as the position of an answer and its answer score, were considered less important. The content of Stack Overflow has been a major source for training large language models. AI-based coding assistants such as GitHub Copilot, which are based on these models, might change the way Stack Overflow is used. However, our findings have implications beyond this specific platform. First, they may help to improve the relevance of community-driven platforms such as Stack Overflow, which provide human advice and explanations of code solutions, complementing AI-based support for software developers. Second, since chat-based AI tools can be prompted to generate code in different ways, knowing which properties influence perceived helpfulness might lead to targeted prompting strategies to generate more readable code snippets.

Problem

• How code comments affect Stack Overflow answer helpfulness
• Comparing block versus inline comments for code comprehension
• Impact of comments on AI-generated code readability

Innovation

• Using block and inline code comments
• Simulating Stack Overflow environment experiment
• Analyzing perceived helpfulness for novices