content

Producing and managing textual and multimedia content and enforcing platform policies by applying automated classifiers, human review workflows, metadata tagging, and tools like Perspective API or custom ML models to detect spam, hate, or copyright violations while managing editorial pipelines and publishing.

content

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

A survey of textual cyber abuse detection using cutting-edge language models and large language models

Jan 09, 2025
JA
J. A. Díaz-García
🏛️ University of Granada | INESC-ID | Instituto Superior Técnico | Universidade de Lisboa

Online textual abuse—including hate speech and cyberbullying—seriously harms users’ mental health and erodes social trust. While large language models (LLMs) enhance detection capabilities, they may also generate harmful content, exacerbating governance challenges. This study systematically reviews text abuse detection methods in Chinese social media and introduces, for the first time, a “technical–ethical” co-analysis framework. We empirically evaluate leading LLMs across four critical dimensions: detection accuracy, bias, robustness, and risk of generating abusive content. By integrating text classification, psychosocial impact modeling, and adversarial generation analysis, we uncover the dialectical role of LLMs—both mitigating and amplifying online abuse. Our findings provide empirically grounded, actionable insights for safe AI governance, including a phased technical roadmap for responsible deployment and mitigation.

Language ModelsOnline MisconductSocial Media

The multi-type, overlapping nature of online hate speech renders conventional binary classification inadequate, motivating the shift toward multi-label classification. Method: We conduct the first systematic review of 46 English-language studies—spanning 28 datasets and 24 models—employing meta-analysis, cross-dataset consistency evaluation, and quantitative assessment of annotation quality (e.g., inter-annotator agreement, IAA). Contribution/Results: We reveal substantial heterogeneity in label taxonomies, dataset sizes, annotation rigor, and evaluation metrics. Key shared challenges include class imbalance, crowdsourcing bias, and sparse minority-label instances. Based on empirical findings, we propose 10 actionable methodological recommendations. We empirically validate the effectiveness of mainstream multi-label architectures—including BERT- and RNN-based models. This work establishes the first academic benchmark and practical guideline for developing robust, comparable, and regulation-compliant multi-label hate speech detection systems.

Analyzing dataset heterogeneity and model evaluation inconsistenciesIdentifying open issues in hate speech classification researchSurveying multi-label hate speech classification models and datasets

Must-Read Papers

Most classic and influential ideas
View more

This study addresses hate speech detection in German news comments—a low-resource, fine-grained content moderation task. We introduce HOCON34k, the first benchmark dataset for this domain, comprising 1,592 manually annotated samples. We conduct the first unified evaluation of GPT-4o, Jigsaw’s Perspective API, and OpenAI’s Moderation API under standardized conditions. Through systematic comparison of zero-, one-, and few-shot prompting strategies, we find that GPT-4o significantly outperforms both commercial APIs on the joint MCC and F2-score metric—achieving approximately a 5-percentage-point gain over the HOCON34k baseline. Our key contributions are threefold: (1) releasing the first publicly available benchmark for hate speech detection in German reader comments; (2) empirically demonstrating the superiority of large language models—and the efficacy of prompt engineering—in low-resource language content moderation; and (3) providing evidence supporting multi-API ensemble approaches for robust, cross-system moderation.

GPT-4Moderation APIPerspective API

Defining, Understanding, and Detecting Online Toxicity: Challenges and Machine Learning Approaches

Sep 13, 2025
GK
Gautam Kishore Shahi
🏛️ University of Duisburg-Essen | Center for Advanced Internet Studies | University of Bochum

During sensitive periods—such as crises and elections—the detection of online harmful content (e.g., hate speech, offensive language) faces core challenges including conceptual ambiguity and poor generalizability across contexts and languages. Method: This study systematically reviews 140 relevant works to clarify definitional boundaries and data limitations; proposes a novel multilingual, cross-platform toxicity detection paradigm; and introduces a comprehensive benchmark dataset covering 32 languages and high-stakes scenarios—including elections and public health emergencies. Leveraging advanced machine learning and NLP techniques, the study optimizes classification models for enhanced cross-lingual and cross-platform robustness. Contribution/Results: The framework significantly improves accuracy and generalizability in toxic content identification, offering a reusable methodological foundation and empirically grounded guidelines for real-world content moderation practices.

Challenges in automated toxicity detection mechanismsDefining and detecting online toxic contentImproving classification models with cross-platform data

This study addresses the surge of AI-generated content on social media platforms and the absence of a systematic governance framework, which raises critical challenges concerning accountability, content disclosure, and regulatory compliance. Through qualitative content analysis, the research systematically examines the AI content governance policies of 40 major social platforms, employing thematic coding and cross-platform comparison to identify six core governance themes. Findings reveal that over two-thirds of platforms have implemented relevant measures, yet most rely on existing content moderation rules, emphasizing enforcement against violations and disclosure requirements while largely neglecting forward-looking issues such as rights attribution and incentive mechanisms. The study proposes the development of a more comprehensive and anticipatory governance framework, complemented by user education tools, to inform platform policy design and regulatory practice.

AI-generated contentcontent governancegenerative AI

To address the growing prevalence of hateful memes on social media, this work investigates zero-shot detection capabilities of vision-language models (VLMs), circumventing the bottleneck of scarce annotated data. We systematically evaluate mainstream VLMs—including CLIP, Flamingo, and LLaVA—via multimodal prompt engineering and employ superpixel occlusion-based interpretability analysis to diagnose misclassification mechanisms. Our key contribution is the first taxonomy of misclassification patterns for zero-shot hateful meme detection, identifying six canonical error types that expose critical robustness deficiencies in VLMs under semantic incongruence, metaphorical abuse, and culturally biased contexts. This taxonomy provides an interpretable, attribution-aware framework for safety alignment and establishes an empirical foundation for designing next-generation content safety guardrails. (149 words)

Assessing zero-shot VLMs for hate meme detectionDeveloping typology for error classification in detectionIdentifying vulnerabilities in VLM-based moderation

This work proposes a unified natural language processing framework to address key challenges in academic integrity, including plagiarism, content fabrication, and authorship verification. The framework integrates four core stylometric tasks: classification of human- versus machine-generated text, distinction between single- and multi-author documents, detection of authorship changes within multi-author texts, and identification of contributing authors in collaborative writing. The study introduces and publicly releases the first academic text dataset generated using Gemini under two distinct instruction settings—standard and strict—and systematically evaluates how prompting strategies affect detection performance. Experimental results demonstrate that texts produced under strict instructions are significantly more adversarial, thereby increasing the difficulty of accurate identification. The code and dataset are made openly available, establishing a new benchmark for research on academic integrity.

academic integrityauthorship verificationmachine-generated text

Latest Papers

What's happening recently
View more

Amid the proliferation of generative models and increasing platform restrictions on behavioral data access, traditional detection methods for malicious information manipulation—relying on content or network features—are becoming increasingly ineffective. This work proposes a platform-agnostic detection framework that, for the first time, models user activity as a sequential decision-making process and identifies manipulative accounts by learning behavioral policies rather than depending on content features. By leveraging behavioral policy as a stable discriminative signal, the approach enables cross-platform, evasion-resistant detection even in environments where content is easily forged and data access is limited. Evaluated on a dataset of 12,064 Reddit users including 99 known Russian IRA accounts, the proposed policy classifier achieves a macro F1-score of 94.9%, substantially outperforming text embedding–based methods while enabling earlier detection and greater robustness.

behavioral policiescoordinated manipulationinformation operations

Keyword-based retrieval in systematic literature reviews (SLRs) incurs high manual screening effort and low precision. Method: This paper proposes a semi-automated screening framework driven by multi-large language model (LLM) consensus. It integrates state-of-the-art open-source and commercial LLMs (2024–2025), employs descriptive prompting for paper classification, generates initial labels via a weighted consensus mechanism, and incorporates human-in-the-loop supervision with real-time correction. A visual interactive tool, LLMSurver, enables human-AI collaborative decision-making. Results: Evaluated on over 8,000 real candidate papers, the framework substantially reduces manual screening workload, achieves lower error rates than individual human experts, and demonstrates that modern open-source LLMs deliver sufficient performance—offering high accuracy, strong interpretability, low cost, and broad applicability.

Automating literature corpus filtration in systematic reviewsEnhancing SLR efficiency through human-supervised AI collaborationReducing manual effort in paper classification using LLMs

Unauthorized use of copyrighted material in large language model (LLM) training data poses escalating legal and ethical challenges. Existing detection methods—such as DE-COP—are computationally intensive and operationally complex, rendering them inaccessible to independent creators. This paper introduces a lightweight, scalable, and open-source copyright detection framework. Building upon DE-COP, it integrates an efficient API invocation strategy, an optimized locality-sensitive hashing (LSH) algorithm for similarity matching, and a modular backend architecture, complemented by an intuitive web frontend. Relative to baseline approaches, the framework achieves 10–30% higher detection efficiency while substantially reducing computational overhead and lowering deployment barriers. The fully open-sourced platform enables creators to autonomously verify whether their works have been incorporated into LLM training corpora, thereby advancing compliance, transparency, and responsible copyright governance in AI development.

Detecting unauthorized copyrighted content in LLM training dataProviding accessible copyright verification tools for independent creatorsReducing computational overhead in existing copyright detection frameworks

Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances

Nov 18, 2025
RK
Rishu Kumar Singh
🏛️ Indian Institute of Technology Patna | National Institute of Technology Patna | Fondazione Bruno Kessler

Existing complaint analysis methods are largely confined to unimodal, short-text inputs (e.g., tweets), rendering them inadequate for complex, multimodal, multi-turn customer support dialogues containing both textual complaints and visual evidence (e.g., screenshots, product images). This work proposes VALOR, the first framework to systematically model such multimodal, multi-turn support conversations for fine-grained joint classification of complaint aspects and severity levels. Its core innovations include: (1) a verification-aware multi-expert reasoning mechanism, (2) a semantic alignment–driven cross-modal fusion strategy, and (3) a chain-of-thought (CoT)–enhanced prompting paradigm. Evaluated on a newly constructed fine-grained multimodal complaint dataset, VALOR significantly outperforms state-of-the-art baselines—particularly under imbalanced text–image distributions—demonstrating strong robustness. This research advances United Nations Sustainable Development Goals SDG 9 (Industry, Innovation and Infrastructure) and SDG 12 (Responsible Consumption and Production).

Analyzing multimodal customer complaints with text and visual evidenceClassifying fine-grained complaint aspects and severity levelsEnsuring semantic coherence between different information modalities

The explosive growth of online video content has rendered manual brand safety review infeasible at scale. This paper presents the first systematic evaluation of multimodal large language models (MLLMs) for multilingual, multi-risk-category video brand safety assessment. We introduce the first fine-grained, high-accuracy multilingual video moderation dataset and benchmark leading MLLMs—including Gemini, GPT, and Llama—against domain-expert human reviewers. Results show that state-of-the-art MLLMs achieve human-level or near-human performance across most risk categories, substantially improving review efficiency and cost-effectiveness. Crucially, we identify prevalent failure modes, including misinterpretation of implicit semantics and cross-modal logical inconsistencies, providing empirically grounded insights for model refinement. The dataset is publicly released to foster reproducible research and industrial deployment in content safety.

Assessing accuracy and cost efficiency of MLLMs in moderationComparing AI and human moderators for brand safety classificationEvaluating MLLMs for multimodal content moderation in videos

Hot Scholars

EC

Eshwar Chandrasekharan

Assistant Professor, University of Illinois Urbana-Champaign
Social ComputingHCIHuman-Centered AIOnline Governance
EF

Emilio Ferrara

Professor of Computer Science at the University of Southern California
Human-Centered AISocial ComputingNetwork ScienceAI Safety
KS

Koustuv Saha

University of Illinois Urbana-Champaign
Computational Social ScienceSocial ComputingHuman-Centered Machine LearningWellbeing
LL

Luca Luceri

Research Assistant Professor @University of Southern California - Information Sciences Institute
Computational Social ScienceNetwork ScienceMachine LearningSocial Media Manipulation
JS

Joseph Seering

School of Computing, KAIST
Social ComputingHuman-Computer InteractionSocial Agents