Linting Style and Substance in READMEs

2026-02-27

AI Summary

This work addresses the challenge of inconsistent README quality, which stems from varying audiences and usage contexts, and the inability of existing tools to simultaneously accommodate style, content, and contextual appropriateness. The paper proposes LintMe, a novel linter that uniquely integrates programmatic rules with large language model (LLM)-based content understanding. LintMe enables users to define context-sensitive checking rules via a lightweight domain-specific language (DSL), combining programmatic validations—such as link verification—with LLM-driven semantic assessments like terminology recognition. This approach enhances documentation quality while preserving authorial autonomy. A user study (N=11) demonstrates that LintMe is both usable and flexible, significantly outperforming baseline approaches that rely solely on direct LLM usage, and its scalability is further validated through illustrative case studies.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

READMEs shape first impressions of software projects, yet what constitutes a good README varies across audiences and contexts. Research software needs reproducibility details, while open-source libraries might prioritize quick-start guides. Through a design probe, LintMe, we explore how linting can be used to improve READMEs given these diverse contexts, aiding style and content issues while preserving authorial agency. Users create context-specific checks using a lightweight DSL that uses a novel combination of programmatic operations (e.g., for broken links) with LLM-based content evaluation (e.g., for detecting jargon), yielding checks that would be challenging for prior linters. Through a user study (N=11), comparison with naive LLM usage, and an extensibility case study, we find that our design is approachable, flexible, and well matched with the needs of this domain. This work opens the door for linting more complex documentation and other culturally mediated text-based documents.

Problem

• README
• linting
• documentation quality
• context-specific checks
• authorial agency

Innovation

• linting
• README
• domain-specific language
• LLM-based evaluation
• documentation quality