Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Stud

2025-12-15

AI Summary

Existing evaluations of AI ethics tools (AIETs) often lack developer-centric perspectives and neglect linguistic and cultural specificity, particularly for under-resourced languages like Portuguese. Method: This study introduces a novel, empirically grounded, developer-oriented evaluation framework—integrating systematic literature review (213 tools), semi-structured interviews with 18 Portuguese-language LLM developers (35 hours total), and cross-tool comparative analysis—to assess four mainstream AIET categories: Model Cards, ALTAI, FactSheets, and Harms Modeling. Contribution/Results: While all four tool types support generic ethical considerations, none effectively identify Portuguese-specific risks—such as idiomatic misinterpretation or culturally embedded biases—and critically lack actionable implementation guidance and localization mechanisms. The study exposes a structural gap between language-specific ethical modeling and engineering deployment, providing empirical evidence and concrete design recommendations for regionally adapted AIETs.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

In Artificial Intelligence (AI), language models have gained significant importance due to the widespread adoption of systems capable of simulating realistic conversations with humans through text generation. Because of their impact on society, developing and deploying these language models must be done responsibly, with attention to their negative impacts and possible harms. In this scenario, the number of AI Ethics Tools (AIETs) publications has recently increased. These AIETs are designed to help developers, companies, governments, and other stakeholders establish trust, transparency, and responsibility with their technologies by bringing accepted values to guide AI's design, development, and use stages. However, many AIETs lack good documentation, examples of use, and proof of their effectiveness in practice. This paper presents a methodology for evaluating AIETs in language models. Our approach involved an extensive literature survey on 213 AIETs, and after applying inclusion and exclusion criteria, we selected four AIETs: Model Cards, ALTAI, FactSheets, and Harms Modeling. For evaluation, we applied AIETs to language models developed for the Portuguese language, conducting 35 hours of interviews with their developers. The evaluation considered the developers' perspective on the AIETs' use and quality in helping to identify ethical considerations about their model. The results suggest that the applied AIETs serve as a guide for formulating general ethical considerations about language models. However, we note that they do not address unique aspects of these models, such as idiomatic expressions. Additionally, these AIETs did not help to identify potential negative impacts of models for the Portuguese language.

Problem

• Evaluates AI ethics tools for language models
• Assesses tools' effectiveness from developers' perspective
• Identifies gaps in addressing language-specific ethical issues

Innovation

• Methodology evaluates AI ethics tools for language models
• Selected four tools through literature survey and criteria
• Conducted developer interviews to assess tool effectiveness