Evaluating Large Language Models for Antisemitic Incident Classification

2026-07-06

AI Summary

Automated identification and fine-grained classification of antisemitic incidents remain an emerging area of research. This study presents the first systematic application of large language models to this task, leveraging a multi-source dataset annotated by domain experts and employing prompt engineering strategies—specifically, incorporating term definitions and contextual examples—to evaluate models such as GPT-4o and Llama-3.2-3B-Instruct. The findings indicate that providing explicit term definitions significantly improves accuracy in identifying rhetorical antisemitic events, whereas contextual examples prove more effective for classifying action-oriented incidents. Overall, GPT-4o demonstrates superior performance, supporting the feasibility of deploying such models for early detection and intervention in real-world settings like campus media.

Abstract

Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful events remains underexplored. We introduce the task of hateful event detection and investigate the ability of AI systems, specifically large language models (LLMs), to discover and classify reports of antisemitic events with fine-grained labels. We evaluate OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on multiple expert-annotated datasets containing antisemitic event descriptions from news articles, civil society reports, and official records. We show that LLMs, particularly GPT-4o, have potential for this task, but substantial improvement is needed. Providing clear term definitions and in-context examples in prompts can improve performance: definitions are most helpful for rhetoric-oriented events (e.g. classical antisemitic tropes), while examples help label action-oriented events (e.g. physical assault). A case study of college newspapers demonstrates that LLMs can help surface relevant real-world events, supporting early monitoring and intervention. Overall, our findings highlight both opportunities and critical gaps in AI's ability to recognize complex harms and underscore the need for collaborative efforts among AI developers, policymakers, and civil society to design models, implement robust evaluation, and develop policy frameworks for defining and combating hate efficiently and effectively.

Problem

• antisemitic incident classification
• hateful event detection
• large language models
• AI for social good
• hate speech monitoring

Innovation

• hateful event detection
• large language models
• antisemitic classification
• prompt engineering
• fine-grained labeling