Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm

2025-05-07

AI Summary

This study reveals how personalization of open-source text-to-image models (e.g., LoRA) exacerbates gendered harms. Through empirical analysis of 40 million images and 230,000 models on CivitAI, we find that 68% of newly uploaded models exhibit NSFW tendencies and 32% enable facial cloning—fueling non-consensual deepfakes and hypersexualized content. Methodologically, we integrate large-scale web crawling, NSFW classification, topic modeling, community behavioral graph analysis, and qualitative coding, while pioneering the systematic application of feminist theory and social construction of technology (SCOT) to generative AI governance. Our contributions comprise a three-tier intervention framework: (1) design reform (e.g., mandatory ethical declarations), (2) dynamic moderation (real-time risk labeling), and (3) platform accountability contracts. This work advances both theoretical understanding and actionable policy for responsible development in open-source AI ecosystems.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Responsible Web: Mitigating online harms, including harassment, spam, and fake reviews

Abstract

Open-source text-to-image (TTI) pipelines have become dominant in the landscape of AI-generated visual content, driven by technological advances that enable users to personalize models through adapters tailored to specific tasks. While personalization methods such as LoRA offer unprecedented creative opportunities, they also facilitate harmful practices, including the generation of non-consensual deepfakes and the amplification of misogynistic or hypersexualized content. This study presents an exploratory sociotechnical analysis of CivitAI, the most active platform for sharing and developing open-source TTI models. Drawing on a dataset of more than 40 million user-generated images and over 230,000 models, we find a disproportionate rise in not-safe-for-work (NSFW) content and a significant number of models intended to mimic real individuals. We also observe a strong influence of internet subcultures on the tools and practices shaping model personalizations and resulting visual media. In response to these findings, we contextualize the emergence of exploitative visual media through feminist and constructivist perspectives on technology, emphasizing how design choices and community dynamics shape platform outcomes. Building on this analysis, we propose interventions aimed at mitigating downstream harm, including improved content moderation, rethinking tool design, and establishing clearer platform policies to promote accountability and consent.

Problem

• Examining how AI model personalization amplifies misogynistic and hypersexualized content
• Analyzing the rise of NSFW content and non-consensual deepfakes in open-source TTI platforms
• Proposing interventions to mitigate harm through better moderation and platform policies

Innovation

• Analyzes CivitAI platform with sociotechnical approach
• Uses dataset of 40M images and 230K models
• Proposes interventions like better content moderation