Leveraging Language Models to Discover Evidence-Based Actions for OSS Sustainability

2026-02-12

AI Summary

This work addresses the limited actionability of existing open-source software (OSS) sustainability prediction models, which often fail to guide maintainers toward concrete interventions. To bridge this gap, we propose the first framework that integrates large language models with empirical software engineering research. Leveraging retrieval-augmented generation (RAG) and a two-stage prompting strategy, our approach automatically extracts 1,922 empirically supported, actionable recommendations—termed ReACTs—from 829 top-tier conference papers (e.g., ICSE, FSE). After rigorous filtering, 1,312 high-quality ReACTs are retained and systematically organized by practice categories. The resulting framework is scalable, reproducible, and interoperable with OSS health assessment tools such as APEX, offering evidence-driven, actionable guidance to support sustainable OSS maintenance.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

When successful, Open Source Software (OSS) projects create enormous value, but most never reach a sustainable state. Recent work has produced accurate models that forecast OSS sustainability, yet these models rarely tell maintainers what to do: their features are often high-level socio-technical signals that are not directly actionable. Decades of empirical software engineering research have accumulated a large but underused body of evidence on concrete practices that improve project health. We close this gap by using LLMs as evidence miners over the SE literature. We design a RAG-pipeline and a two-layer prompting strategy that extract researched actionables (ReACTs): concise, evidence-linked recommendations mapping to specific OSS practices. In the first layer, we systematically explore open LLMs and prompting techniques, selecting the best-performing combination to derive candidate ReACTs from 829 ICSE and FSE papers. In the second layer, we apply follow-up prompting to filter hallucinations, extract impact and evidence, and assess soundness and precision. Our pipeline yields 1,922 ReACTs, of which 1,312 pass strict quality criteria and are organized into practice-oriented categories connectable to project signals from tools like APEX. The result is a reproducible, scalable approach turning scattered research findings into structured, evidence-based actions guiding OSS projects toward sustainability.

Problem

• Open Source Software
• Sustainability
• Actionable Recommendations
• Evidence-Based Practices
• Software Engineering

Innovation

• Large Language Models
• Evidence-Based Actions
• Open Source Software Sustainability
• Retrieval-Augmented Generation
• Actionable Recommendations