Bye Bye Perspective API: Lessons for Measurement Infrastructure in NLP, CSS and LLM Evaluation

2026-04-28

AI Summary

This study addresses the critical threats to scientific reliability in natural language processing, computational social science, and large model evaluation posed by overreliance on closed-source toxicity detection tools such as Perspective API. Such dependence undermines reproducibility, version control, and conceptual pluralism in measuring toxic content. The work systematically exposes the resulting epistemic and infrastructural vulnerabilities and proposes a novel open-source framework for assessing toxicity and hate speech that is reproducible, adaptable, and grounded in measurement theory. By integrating versioned model deployment and multi-stakeholder governance mechanisms, this framework offers a transparent and sustainable alternative to proprietary APIs, aiming to shift the research community toward accountable evaluation practices and avoid repeating the pitfalls of dependency on closed large models.

Abstract

The closure of Perspective API at the end of 2026 discards what has functioned as the de facto standard for automated toxicity measurement in NLP, CSS, and LLM evaluation research. We document the structural dependence that the communities built on this single proprietary tool and discuss how this dependence caused epistemic problems that have affected - and will likely continue to affect - collective research efforts. Perspective's model was periodically updated without versioning or disclosure, its annotation structure reflected a single corporate operationalisation of a contested concept, and its scores were used simultaneously as an evaluation target and an evaluation standard. Its closure leaves behind non-updatable benchmarks, irreproducible results, and ultimately a field at risk of perpetuating these issues by turning to closed-source LLMs. We use Perspective's announced termination as an opportunity to call for an independent, valid, adaptable, and reproducible toxicity and hate speech measurement infrastructure, with the technical and governance requirements outlined in this paper.

Problem

• toxicity measurement
• measurement infrastructure
• Perspective API
• reproducibility
• epistemic dependence

Innovation

• measurement infrastructure
• toxicity evaluation
• reproducibility
• open benchmarking
• LLM evaluation