🤖 AI Summary
This work addresses prevalent methodological flaws in scientific Python code—such as data leakage and improper cross-validation—that often yield seemingly valid but erroneous conclusions and are difficult to detect with conventional tools. To tackle this, the authors propose a two-tier automated detection framework: a large language model (LLM) generates detection patterns without reliance on handcrafted rules or specific library versions, while a lightweight local model performs efficient runtime checks. This approach requires only a minimal number of tokens to adapt to new library versions, substantially enhancing scalability and maintainability. Empirical evaluation demonstrates high effectiveness, achieving precision rates of 65% on Kaggle notebooks, 62% on AI/ML research code, and 97.7% in controlled tests, with recall consistently reaching 100%.
📝 Abstract
Methodology bugs in scientific Python code produce plausible but incorrect results that traditional linters and static analysis tools cannot detect. Several research groups have built ML-specific linters, demonstrating that detection is feasible. Yet these tools share a sustainability problem: dependency on specific pylint or Python versions, limited packaging, and reliance on manual engineering for every new pattern. As AI-generated code increases the volume of scientific software, the need for automated methodology checking (such as detecting data leakage, incorrect cross-validation, and missing random seeds) grows. We present scicode-lint, whose two-tier architecture separates pattern design (frontier models at build time) from execution (small local model at runtime). Patterns are generated, not hand-coded; adapting to new library versions costs tokens, not engineering hours. On Kaggle notebooks with human-labeled ground truth, preprocessing leakage detection reaches 65% precision at 100% recall; on 38 published scientific papers applying AI/ML, precision is 62% (LLM-judged) with substantial variation across pattern categories; on a held-out paper set, precision is 54%. On controlled tests, scicode-lint achieves 97.7% accuracy across 66 patterns.