COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act

2024-10-10

arXiv.org

AI Summary

The EU AI Act lacks actionable technical standards, impeding rigorous compliance assessment of large language models (LLMs). Method: We propose the first technical interpretation framework and open-source evaluation suite aligned with the Act, systematically mapping regulatory requirements to quantifiable technical metrics—robustness, safety, fairness, and diversity. We introduce the first Act-aligned, multidimensional LLM benchmarking suite and design a modular, extensible automated evaluation framework grounded in literature review and state-of-the-art methodologies to support compliance gap analysis. Contribution/Results: Empirical evaluation across 12 mainstream LLMs reveals significant deficiencies across multiple critical compliance dimensions. Our framework provides empirically grounded, actionable guidance for both regulatory enforcement and model development, bridging the gap between high-level legislation and implementable technical assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: AI for Accessibility

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.

Problem

• AI Regulation
• Language Models
• Compliance Evaluation

Innovation

• COMPL-AI Framework
• Legal-to-Technical Translation
• Model Compliance Evaluation