JETHICS: Japanese Ethics Understanding Evaluation Dataset

2025-06-19

AI Summary

Existing AI models lack systematic evaluation benchmarks for Japanese ethical reasoning. To address this gap, we introduce JETHICS—the first comprehensive Japanese ethical reasoning benchmark, comprising 78,000 human-annotated instances spanning four major normative ethical theories and commonsense moral judgment. JETHICS represents the first high-fidelity localization of the English ETHICS framework to Japanese, grounded in theory-driven, multidimensional taxonomy design and rigorous manual annotation. It employs a standardized zero-shot prompting protocol, ensuring compatibility with both open- and closed-weight large language models. Experimental results show that GPT-4o achieves a mean score of 0.7, while the strongest open-weight Japanese LLM scores only ~0.5—highlighting substantial room for improvement. By establishing a culturally grounded, linguistically accurate evaluation infrastructure, JETHICS fills a critical void in non-English AI ethics assessment and enables rigorous cross-cultural value alignment research.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In this work, we propose JETHICS, a Japanese dataset for evaluating ethics understanding of AI models. JETHICS contains 78K examples and is built by following the construction methods of the existing English ETHICS dataset. It includes four categories based normative theories and concepts from ethics and political philosophy; and one representing commonsense morality. Our evaluation experiments on non-proprietary large language models (LLMs) and on GPT-4o reveal that even GPT-4o achieves only an average score of about 0.7, while the best-performing Japanese LLM attains around 0.5, indicating a relatively large room for improvement in current LLMs.

Problem

• Evaluating ethics understanding of AI models in Japanese
• Creating a Japanese dataset based on normative ethical theories
• Assessing performance gaps in current large language models

Innovation

• Japanese ethics dataset JETHICS with 78K examples
• Based on normative theories and commonsense morality
• Evaluates LLMs including GPT-4o and Japanese models