ACL Ready: RAG Based Assistant for the ACL Checklist

2024-08-07

arXiv.org

AI Summary

Self-reported ethical checklists often suffer from inaccurate completion due to authors’ misinterpretation of paper content, diminishing their effectiveness in prompting early reflection on ethics, societal impact, and reproducibility. To address this, we introduce the first retrieval-augmented AI assistant specifically designed for the ACL Responsible Research Checklist. Our approach innovatively adapts RAG to academic ethical self-assessment: it jointly indexes ACL paper metadata and official guidelines into a vector store to enable semantic alignment between checklist items and manuscript content, and leverages a fine-tuned language model to generate structured, context-aware feedback. A user study with 13 participants demonstrated that 92% rated the tool as practical and easy to use, and 77% successfully retrieved their intended information. The implementation is publicly available under the CC BY-NC 4.0 license.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Algorithmic accountability and transparency on the web
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

The ARR Responsible NLP Research checklist website states that the"checklist is designed to encourage best practices for responsible research, addressing issues of research ethics, societal impact and reproducibility."Answering the questions is an opportunity for authors to reflect on their work and make sure any shared scientific assets follow best practices. Ideally, considering the checklist before submission can favorably impact the writing of a research paper. However, the checklist is often filled out at the last moment. In this work, we introduce ACLReady, a retrieval-augmented language model application that can be used to empower authors to reflect on their work and assist authors with the ACL checklist. To test the effectiveness of the system, we conducted a qualitative study with 13 users which shows that 92% of users found the application useful and easy to use as well as 77% of the users found that the application provided the information they expected. Our code is publicly available under the CC BY-NC 4.0 license on GitHub.

Problem

• Assisting authors with conference checklist responses
• Evaluating RAG systems for checklist answer generation
• Analyzing problems in human-authored checklist answers

Innovation

• RAG-based assistant for conference checklists
• Dataset of 1975 ACL checklist responses
• Benchmarks RAG and LLM systems