trust and safety

The proliferation of toxic text generated by large language models (LLMs) undermines the robustness of toxicity classifiers and increases their susceptibility to adversarial attacks. Method: This paper proposes a mechanistic interpretability–driven active defense framework: it introduces attention-head-level circuit analysis—the first such application—to diagnose classifier vulnerabilities; integrates fine-grained attribution with adversarial attack localization to identify critical, attack-prone components; and enhances robustness via targeted circuit suppression. Contribution/Results: Evaluated on BERT and RoBERTa architectures across diverse demographic datasets, the method significantly improves classification accuracy under adversarial perturbations. It further uncovers systematic differences in model vulnerability across demographic groups, revealing fairness-related failure modes. By unifying interpretability, robustness, and fairness, this work establishes a novel paradigm for building trustworthy, auditable, and attack-resilient content moderation systems.

This work addresses the limitations of existing abuse detection methods, which rely on static models and manual annotations and struggle to handle dynamic, context-sensitive online abusive behaviors. It proposes the first large language model (LLM)-integrated framework spanning the entire lifecycle of abuse detection, systematically encompassing label and feature generation, detection, appeal review, and audit governance. Bridging academic research and industrial practice, the study explores LLMs’ capabilities in contextual reasoning, policy interpretation, and cross-modal understanding. The authors delineate key architectural considerations for each phase, evaluate LLMs’ potential and limitations regarding interpretability, policy alignment, and multimodal fusion, and identify critical challenges—including latency, cost, determinism, adversarial robustness, and fairness—thereby offering a principled direction toward building reliable, accountable, large-scale abuse governance systems.

This study addresses the critical misalignment between content moderation policies and user experience in generative AI (GAI) products. Employing qualitative content analysis, discourse mining from Reddit communities, and comparative policy analysis across 14 mainstream GAI tools, we systematically identify three core tensions: severe deficits in policy transparency, widespread absence of user appeal and participatory mechanisms, and lack of explainability in moderation decisions. While automated moderation systems effectively block malicious content, frequent false positives, inadequate feedback channels, and ineffective support erode user trust and satisfaction, provoking widespread frustration. Our findings provide empirical grounding for GAI governance and propose a user-centered framework to enhance moderation transparency, appealability, and explainability—thereby bridging a key gap in human-AI collaborative moderation research focused on end-user experience.

Although state-of-the-art large language models employ output-level safety mechanisms, they may still inadvertently leak harmful knowledge through indirect prompting, enabling open-source models—after fine-tuning—to reconstruct hazardous capabilities, thereby posing ecosystem-level risks. This work presents the first systematic investigation of such cross-model capability transfer threats and introduces a three-stage elicitation attack framework: by crafting benign prompts that are semantically proximate to harmful tasks, adversaries can extract implicit hazardous information from safeguarded models and use it to fine-tune open-source counterparts. Experiments on dangerous chemical synthesis tasks demonstrate that this approach recovers approximately 40% of the performance gap between protected and unrestricted models, with attack efficacy significantly amplified by both the capability of the frontier model and the scale of fine-tuning data, thereby challenging the adequacy of current safety paradigms.

Existing social media sensitive content detection tools suffer from limited customizability, narrow category coverage—particularly lacking long-tail classes such as drug-related and self-harm content—high privacy risks, and the absence of a unified evaluation benchmark. To address these issues, this work introduces the first high-quality, uniformly annotated dataset covering six sensitive content categories: conflict language, abuse, pornography, drug-related content, self-harm, and spam. We establish standardized protocols for data collection and human annotation. Leveraging this dataset, we supervise fine-tuning of open-source large language models (e.g., LLaMA) and design a comprehensive, multi-dimensional evaluation benchmark. Experimental results demonstrate that our approach consistently outperforms both the LLaMA baseline and the OpenAI API across all six detection tasks, achieving average improvements of 10–15%. Gains are especially pronounced for scarce categories (e.g., drug-related and self-harm content), validating the effectiveness and deployability of open-source LLM fine-tuning for fine-grained sensitive content identification.

This work addresses the limitations of traditional content moderation systems, which rely on centralized rules and fail to account for users’ subjective sensitivities to harmful content. The authors propose a novel personalized reasoning framework based on large language models that integrates a multi-agent architecture with user sensitivity modeling. By simulating collaborative interactions among expert, moderator, and user-profile agents, the system tailors content filtering to individual preferences. The approach significantly enhances alignment between moderation outcomes and users’ personal sensitivity thresholds while maintaining platform-level governance efficacy. Experimental results demonstrate up to a 32% improvement in accuracy over non-personalized baselines, offering a scalable new paradigm for content moderation that balances individual rights with platform responsibilities.

This work addresses the high cost of manually drafting and maintaining domain-specific content safety policies by proposing Deep Policy Research (DPR), a system that requires only a small set of seed policies per domain. DPR employs a specialized structured research loop that integrates a single web search, a lightweight agent framework, and a compact reading-focused large language model to iteratively generate queries, distill relevant web content, and synthesize it into structured policy rules. Compared to general-purpose deep research systems, DPR demonstrates significant improvements in both efficiency and effectiveness, outperforming definition-only and in-context learning baselines on the OpenAI inappropriate content benchmark as well as an internal multimodal ad moderation benchmark, achieving expert-level policy quality in several domains.

Widespread reliance on unlicensed web-scale data for AI pretraining poses escalating global copyright infringement risks, yet current regulatory approaches remain predominantly reactive, lacking proactive, pre-training compliance mechanisms. Method: We systematically analyze copyright governance frameworks across the EU, U.S., and major Asia-Pacific jurisdictions, identifying structural deficiencies in licensing acquisition, content filtering, and enforcement oversight. To address delayed risk detection and incomplete technical coverage during pretraining, we propose a “Proactive Multi-Layer Filtering Framework” integrating access control, perceptual hashing, ML-based classifiers, dynamic database matching, and transparency tools. Contribution/Results: The framework enables end-to-end identification, blocking, and verifiable mitigation of copyright risks in training data pipelines. It is the first to embed copyright compliance intrinsically into the AI training frontend—establishing a technically feasible, governance-aligned pathway that balances creator rights with sustainable AI development.

This work addresses the privacy risks of safety classifiers trained on sensitive data involving self-harm and mental health, which are vulnerable to membership inference attacks that can leak user information. The authors propose a boundary-targeting strategy that, for the first time, integrates low-confidence samples with membership inference attacks to expose how models rely on memorization rather than generalization when handling ambiguous inputs, thereby amplifying membership signals. Experimental results demonstrate that, at a 5% false positive rate, the method successfully recovers 19% of conversations labeled as indicating emotional distress—achieving 3.5 times the performance of the current state-of-the-art approach. The study further validates that adding noise effectively mitigates this privacy vulnerability.