The 2024 Foundation Model Transparency Index

2024-07-17

AI Summary

Foundational models suffer from systemic transparency deficits across critical dimensions—including copyright status, data provenance, labor practices in training data curation, and societal impact. Method: This study introduces a standardized transparency assessment framework comprising 100 metrics, featuring—for the first time—a developer-coordinated submission and verification protocol enabling longitudinal, cross-year comparison on consistent indicators. Building upon the 2023 baseline, the methodology advances to structured reporting collection and mixed qualitative-quantitative evaluation. Results: Assessing 14 leading AI developers, the average score rose to 58/100 (+21 points), with each provider disclosing an average of 16.6 previously non-public items; all 14 reports are publicly accessible. Key contributions include: (1) establishing the first industry-collaborative transparency assessment paradigm; (2) identifying persistent blind spots in copyright attribution, data labor practices, and downstream societal impacts; and (3) delivering a reproducible, traceable empirical benchmark for AI governance.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Interpretability, Explainability, and Transparency
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Algorithmic accountability and transparency on the web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Foundation models are increasingly consequential yet extremely opaque. To characterize the status quo, the Foundation Model Transparency Index (FMTI) was launched in October 2023 to measure the transparency of leading foundation model developers. FMTI 2023 assessed 10 major foundation model developers (e.g. OpenAI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study after 6 months: we score 14 developers against the same 100 indicators. While in FMTI 2023 we searched for publicly available information, in FMTI 2024 developers submit reports on the 100 transparency indicators, potentially including information that was not previously public. We find that developers now score 58 out of 100 on average, a 21 point improvement over FMTI 2023. Much of this increase is driven by developers disclosing information during the FMTI 2024 process: on average, developers disclosed information related to 16.6 indicators that was not previously public. We observe regions of sustained (i.e. across 2023 and 2024) and systemic (i.e. across most or all developers) opacity such as on copyright status, data access, data labor, and downstream impact. We publish transparency reports for each developer that consolidate information disclosures: these reports are based on the information disclosed to us via developers. Our findings demonstrate that transparency can be improved in this nascent ecosystem, the Foundation Model Transparency Index likely contributes to these improvements, and policymakers should consider interventions in areas where transparency has not improved.

Problem

• Measures transparency of foundation model developers.
• Assesses 14 developers on 100 transparency indicators.
• Identifies sustained opacity in copyright, data access, and labor.

Innovation

• Developers submit reports on 100 transparency indicators
• Average score improved from 37 to 58 out of 100
• Transparency reports consolidate disclosed information for each developer